* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Llenceu aquesta pàgina i substituïu-la per aquella que us faciliti... tat d’Informació i Projecció Institucionals (UIPI), disponible al formulari
Survey
Document related concepts
Embodied cognitive science wikipedia , lookup
Biological neuron model wikipedia , lookup
Eyeblink conditioning wikipedia , lookup
Apical dendrite wikipedia , lookup
Holonomic brain theory wikipedia , lookup
Neuropsychopharmacology wikipedia , lookup
Optogenetics wikipedia , lookup
Limbic system wikipedia , lookup
Development of the nervous system wikipedia , lookup
Subventricular zone wikipedia , lookup
Spatial memory wikipedia , lookup
Hippocampus wikipedia , lookup
Neuroanatomy of memory wikipedia , lookup
Transcript
Llenceu aquesta pàgina i substituïu-la per aquella que us faciliti la Unitat d’Informació i Projecció Institucionals (UIPI), disponible al formulari següent http://www.upf.edu/uii/sgrafics/formulari_tesi.htm The hippocampus code A computational study of the structure and function of the hippocampus. César Rennó Costa Tesi Doctoral UPF / 2012 Supervisada pel Dr. Paul Verschure Synthetic, Perceptive, Emotive and Cognitive Systems (SPECS) and Institució Catalana de Recerca i Estudis Avançats (ICREA). ... By César Rennó Costa, 2012 Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported You are free to Share – to copy, distribute and transmit the work Under the following conditions: • Attribution – You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). • Noncommercial – You may not use this work for commercial purposes. • No Derivative Works – You may not alter, transform, or build upon this work. With the understanding that: Waiver – Any of the above conditions can be waived if you get permission from the copyright holder. Public Domain – Where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license. Other Rights – In no way are any of the following rights affected by the license: • Your fair dealing or fair use rights, or other applicable copyright exceptions and limitations; • The author’s moral rights; • Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights. Notice – For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to this web page. The court’s PhD was appointed by the rector of the Universitat Pompeu Fabra on .............................................., 2012. Chairman Member Secretary The doctoral defense was held on ......................................................., 2012, at the Universitat Pompeu Fabra and scored as ................................................... PRESIDENT MEMBERS SECRETARY For my beloved wife Agraïments My biggest thanks to ... ... the city of Barcelona. Place where I had the most amazing experiences and met the most interesting people. A special thanks to the sun, to the beach and to the good catalan food. ... my mother, who always trusted and encouraged my decisions. Her care and support have been truly fundamental for everything that I accomplished so far. ... my father, my role model. Who taught me that things are just simple. ... my wife for boarding with me in this journey overseas. ... the twin brother André Luvizotto for being my wingman, from Brazil to Thailand. And also to Sassi! ... my supervisor Paul Verschure for introducing me to the neuroscience and guiding through the whole higher-education experience. ... my mentor John Lisman, whose patience, availability and clarity of thought were genuinely inspiring. ... Jonatas Manzolli and Fernando Von Zuben for introducing me to science and research and opening me so many opportunities. ... my brother and sister-in-law, for keeping the surfin’ dreams alive. ... my great culé friends Anna-Felipe and Aisha-José. ... all my Costa family, friends and in-laws in Belo Horizonte. ... all my family in Rio de Janeiro. In special to my dear goddaughter Sophia. iv v ... all my Rennó family that is all around. In special to my dear grandmother. ... all the friends from Brazil who came to visit me and all of those that couldn’t. ... my friends from the pachanga de domingo y los Pachanga All-Stars. ... The SPECS team: Anna Mura, Armin Duff, Marti Sanchez Fibla, Sylvain Le Groux, Xerxes Arsiwalla, Alberto Betella, Belén Rubio Ballester, Encarni Marcos, Ivan Herreros Alonso (special thanks for translating my abstract to catalan), Luis Bobo, Riccardo Zucca, Vicky Vouloutsi, Michelle Wilson, Manuel Ebert, Jens Nirme, Fotios Balampanis, Eliza-Nefeli Tsaoussis, Diogo Pata, Daniel Pacheco Estefan, Sytse Wierenga, Pedro Omedas, Enrique Martínez, Alex Escuredo Chimeno, Christian Moscardi, Mireia Mora, Carme Buisan. Ulysses Bernardet, Martin Inderbitzin, Zenon Mathews, Elena Kokkinara, Fabio Rotondi, Arnau Espinosa, Cristina Campillo, Melle Hofman, Hannu Markus Järvinen, Sergi Bermúdez, Andrea Giovannucci, Mónica Cameirão, Alexander Valjamae, Jose Maria Blanco Calvo, Dor Konforty, Miguel Lechón, Noemí Conesa, Ana Pesquita, Anant Dhir, Fabio Manzi, Ninuska. ... the DTIC and UPF staff. ... the PhD committee. ... Google Inc., Microsoft Gaming Division, Sony Entertainment, Ubuntu, EA Sports, Infinity Ward, Mendeley, Mathworks and many other companies that, for a little price, made my life much easier. Abstract There is no consensual understanding on what the activity of the hippocampus neurons represents. While experiments with humans foster a dominant view of an episodic memory system, experiments with rodents promote its role as a spatial cognitive system. Although there is abundant evidence pointing to an overlap between these two theories, the dissociation is sustained by conflicting physiological data. This thesis proposes that the functional role of the hippocampus should be analyzed in terms of its structure and function rather than by the correlation of neuronal activity and behavioral performance. The identification of the hippocampus code, i.e. the set of computational principles underlying the input-output transformations of neural activity, might ultimately provide a unifying understanding of its role. In this thesis we present a theoretical model that quantitatively describes and interprets the selectivity of regions of the hippocampus to spatial and non-spatial variables observed in experiments with rats. The results suggest that the multiple aspects of memory expressed in human and rodent data are derived form similar principles. This approach suggests new principles for memory, pattern completion and plasticity. In addition, by creating a causal tie between the neural circuitry and behavior through a robotic control framework we show that the conjunctive nature of neural selectivity observed in the hippocampus is needed for effective problem solving in realworld tasks such as foraging. Altogether, these results advance the concept that the hippocampal code is generic to the different aspects of memory highlighted in the literature. viii Resum Actualment, no hi ha consens científic respecte a la informació representada en la activitat de les célules del hipocamp. D’una banda, experiments amb humans sostenen una visión de la funció de l’hipocamp com a un sistema per l’emmagatzematge de memóries episódiques, mentre que la recerca amb rodents enfatitza una visió com a sistema cognitiu espacial. Tot i que existeix abundant evidència experimental que indica una possible sobreposició d’ambdues teories, aquesta dissociació també es manté en part en base a dades fisiològiques aparentment incompatibles. Aquesta tèsi poposa que l’hippocamp té un rol funcional que s’hauría d’analitzar en termes de la seva estructura i funció, enlloc de mitjança estudis correlació entre activitat neuronal i comportament. La identificació d’un codi a l’hipocamp, es a dir, el conjunt de principis computacionals que conformen les transformacions d’entrada i sortida de l’activitat neuronal, hauría de proporcionar un explicació unificada de la seva funció. En aquesta tèsi presentem un model teòric que descriu quantitativament i que interpreta la selectivitat de certes regions de l’hipocamp en funció de variables espaials i no-espaials, tal i com observada en experiments amb rates. Aquest resultat suggereix que multiples aspectes de la memòria expressada en humans i rodents deriven d’uns mateixos principis. Per aquest motius, proposem nous principis per la memòria, l’auto-completat de patrons i plasticitat. A més, mitjançant aplicacions robòtiques, creem d’un nexe causal entre el circuit neural i el comportament amb el que demostrem la naturalesa conjuntiva de la selectivitat neuronal observada en el hipocamp es necessària per la solució de problemes pràctics comuns, com per example la cerca d’aliments. Tot plegat, aquests resultats avancen en l’idea general de que el codi de l’hipocamp es genèric i aplicable als diversos tipus de memòries estudiades en la literatura. x Publications Included in the thesis Peer-reviewed Rennó-Costa, C., Lisman, J. E., & Verschure, P. F. M. J. (2010a). The mechanism of rate remapping in the dentate gyrus. Neuron, 68(6):1051–8. ISSN 1097-4199. doi: 10.1016/j.neuron.2010.11.024 Rennó-Costa, C., Luvizotto, A., Marcos, E., Duff, A., Sanchez-Fibla, M., & Verschure, P. F. M. J. (2011). Integrating neuroscience-based models towards an autonomous biomimetic Synthetic Forager. Dins 2011 IEEE International Conference on Robotics and Biomimetics, ps. 210–215. IEEE, Phuket, Thailand. ISBN 978-1-4577-2138-0. doi: 10.1109/ROBIO.2011. 6181287 Rennó-Costa, C., Luvizotto, A., Betella, A., Sanchez Fibla, M., & Verschure, P. F. M. J. (2012c). Internal drive regulation of sensorimotor reflexes in the control of a catering assistant autonomous robot. Dins Lecture Notes in Artificial Intelligence: Living Machines In preparation Rennó-Costa, C., Lisman, J. E., & Verschure, P. F. M. J. (2012a). The mechanism of attractor dynamics in the CA3. In Preparation Rennó-Costa, C. & Verschure, P. F. M. J. (2012). Nonspatial selectivity of place cells supports quasi-optimal behavior in mixed spatial/nonspatial tasks. In Preparation xii xiii Other publications and abstracts Duff, A., Rennó-Costa, C., Marcos, E., Luvizotto, A., Giovannucci, A., Sanchez-Fibla, M., Bernardet, U., & Verschure, P. (2010). From Motor Learning to Interaction Learning in Robots, volum 264 de Studies in Computational Intelligence. Springer Berlin Heidelberg, Berlin, Heidelberg. ISBN 978-3-642-05180-7. doi: 10.1007/978-3-642-05181-4 Luvizotto, A., Rennó-Costa, C., Pattacini, U., & Verschure, P. F. M. J. (2011). The encoding of complex visual stimuli by a canonical model of the primary visual cortex: Temporal population code for face recognition on the iCub robot. Dins 2011 IEEE International Conference on Robotics and Biomimetics, ps. 313–318. IEEE, Phuket, Thailand. ISBN 9781457721373. doi: 10.1109/ROBIO.2011.6181304 Luvizotto, A., Rennó-Costa, C., & Verschure, P. F. M. J. (2012). A waveletbased neural model to optimize and read out a temporal population code. Frontiers in computational neuroscience, 6:21. ISSN 1662-5188. doi: 10. 3389/fncom.2012.00021 Rennó-Costa, C., Lisman, J. E., & Verschure, P. F. M. J. (2010b). The mechanism of rate remapping in the dentate gyrus. Dins Society for Neuroscience Abstracts. San Diego, CA, USA Rennó-Costa, C., Lisman, J. E., & Verschure, P. F. M. J. (2012b). The mechanism of attractor dynamics in the CA3. Dins Society for Neuroscience Abstracts. New Orleans, LA, USA Contents Agraïments iv Abstract viii Resum x Publications xii Contents xv List of Figures xvii 1 Introduction 2 The 2.1 2.2 2.3 2 hippocampus 8 Place cells . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Other spatially driven cells . . . . . . . . . . . . . . . . . . . 18 Conjunctive place cells and rate remapping . . . . . . . . . . 21 3 Computational models of the medial temporal lobe 3.1 Computational models of spatial selectivity . . . . . . 3.1.1 Computational models of grid cells . . . . . . . 3.1.2 Computational models of place cells . . . . . . 3.1.3 Place cell navigation and behavior . . . . . . . 3.2 Computational models of memory . . . . . . . . . . . 3.2.1 Pattern completion and attractor dynamics . . 3.2.2 Memory sequences in the hippocampus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 24 25 31 38 39 39 40 4 Mechanisms of conjunctive selectivity in the dentate gyrus 42 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 xv xvi CONTENTS 4.3 4.4 4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . Supplemental material . . . . . . . . . . . . . . . . . . . . . 4.4.1 Alternative assumptions about how the LEC responds to morphing . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Differences in how DG and LEC encode sensory information . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Comparison to other models . . . . . . . . . . . . . . Experimental procedures . . . . . . . . . . . . . . . . . . . . . 52 . 58 . 58 . 59 . 59 . 61 5 Mechanisms of conjunctive selectivity in the CA3 72 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6 Mechanisms of hippocampal behavioral control 6.1 Theoretical study on behavior . . . . . . . . . . . 6.1.1 Introduction . . . . . . . . . . . . . . . . 6.1.2 Results . . . . . . . . . . . . . . . . . . . 6.1.3 Discussion . . . . . . . . . . . . . . . . . . 6.1.4 Methods . . . . . . . . . . . . . . . . . . . 6.2 Robot experimentation . . . . . . . . . . . . . . . 6.2.1 Introduction . . . . . . . . . . . . . . . . 6.2.2 Results . . . . . . . . . . . . . . . . . . . 6.2.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Conclusion A Internal drive regulation of sensorimotor reflexes A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . A.2 The control architecture . . . . . . . . . . . . . . . A.2.1 The hardware . . . . . . . . . . . . . . . . . A.2.2 The autonomous control system . . . . . . A.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . A.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . Bibliography 84 84 85 91 94 96 97 98 105 114 116 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 121 124 124 125 131 135 138 List of Figures 2.1 2.2 2.3 2.4 The medial temporal lobe in the brain. (top) Coronal cut of whole-brain Macaca mulatta (bottom) Nissl-sagittal cut of whole-brain Rattus Norvegicus. Highlighted areas: dentate gyrus (DG), CA1, CA3, subiculum (Sub), entorhinal cortex (EC, lateral LEC, medial MEC), and perirhinal cortex (36). Adapted from brainmaps.org. . . . . . . . . . . . . . . . . . . . . . . . . Hippocampus anatomy. Drawing of the neural circuitry of a rodent hippocampus by Ramón y Cajal (1909). Diagram with most relevant connections within the hippocampus and the entorhinal cortex. . . . . . . . . . . . . . . . . . . . . . . . . . . . Illustration of the methods for the recording of place cells. (A) Recording setup. Rat with implanted immutable drive with multiple tetrodes is free to move inside an arena. A camera is used to track the position of the rat. Tracking and neurophysiological data are time stamped. (B) Common place cell representations. (top) Spikes (red dots) overlying the trajectory of the rat during one recording session. (down) Rate map of the spatial activity. Frequency varies from highest (red) to silent (blue). . . . . . . . . . . . . . . . . . . . . . . . . . . . . Illustration of the spatial proprieties of place cells. (A) Rotation of dominant environmental cues (such as a white card) causes similar transformation in the place cells. (B) Changes in the dimensions of the environment causes similar transformation in the place cells. (C) Different arenas (represented with black and gray border) with the same dimensions cause place cells to exhibit uncorrelated spatial activity. (D) Place cells are stable in the dark if the rat had experienced the environment with light previously. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 . 10 . 15 . 17 xvii xviii LIST OF FIGURES 2.5 Illustration of other spatially selective cells. (A) Rate map of a grid cell with triangular grid organization. (B) Polar plot of the angular response of a head direction cell. (C) Rate map of a border cell in an arena with an internal wall. . . . . . . . . . . . 18 2.6 Illustration of the spatial variations of grid cells. (A) Scale or intervertex distance: the distance between place fields in the same triangular formation. (B) Angular phase: the angle between place fields in the same triangular formation. (C) Spatial phase: the absolute position of a specific place field in the xand y-axis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.7 Illustration of the rate remapping phenomenon. (A) Different arena shapes that provoke gradual rate remapping and illustrative rate maps for CA3 (B) and DG (C). Red refers to high rate while deep blue refers to silent. Graph at right is the mean fire rate of the place fields according to the morphing phase. For DG, upper place field (black) and bottom place field (gray) are represented. (D) Procedure to compute the population vector (PV). The rate of all cells at a specific position are included in sequence in an array. The activity ensemble change is computed by the mean correlation of the PV of two conditions computed for every position. (E) Illustration of the PV correlation curve for DG and CA3. CA3 present no difference between 1-step morphing and two successive recordings of the same arena. Adapted from (Leutgeb et al., 2007). . . . . . . . . . . . . . . . . . . . . . 22 3.1 Illustration of the oscillatory interference model. (A) Each grid cell has three dendritic subunits that receives input from different head-direction cells with phase intervals of 60. The dendritic subunits have an intrinsic internal oscillation that is modulated by the integral of the speed input of the headdirection cells. (B) A grid cells exhibit theta based activity when the three sub-units have similar frequencies that causes an oscillatory interference. The movement of the animal changes the frequency of each dendritic subunit leading to successive active and silent states that give rise to the grid pattern. . . . . . . . . 26 LIST OF FIGURES xix 3.2 Illustration of the intrinsic persistent spiking model. (A) Each grid cell receives input of three different head direction cells with phase intervals of 60. (B) A grid cells is active when the persistent firing of three head direction cells are synchronized. The movement of the animal changes the phase of each head direction cell leading to successive synchronization and dissynchronization states that give rise to the grid pattern. . . . . . 28 3.3 Illustration of the continuous attractor dynamic models. Bump of activity (black with high activity and white with low activity) in the bidimensional topological organization of the entorhinal cortex neural network with (A) rectangular neighborhood (McNaughton et al., 2006) and (B) triangular neighborhood (Guanella & Verschure, 2006). In both models the cells in the boundaries are interconnected allowing the formation of periodic place fields. (C) Bump is maintained by homogeneous lateral connectivity (shown in a linear representation for clarity). (D) Bump of activity is moved by the modulation of the lateral connectivity in the direction of the animal motion. . . . . . . . . 29 3.4 Illustration of the place cell model based on border cells. (A) Example of the typical rate map of a BVC (boundary vector cell) in three environments. (B) The activity of a place cells is obtained by the integration (followed by a linear threshold filter) of multiples BVC cells. . . . . . . . . . . . . . . . . . . . . . . . . 32 3.5 Illustration of how place cells are generated from grid cells in the integrative-competitive model. (top) The integration of multiple grid cells will generate a spatial dependent excitation for a granule cell (bottom-left) . The E%-MAX winnertake-all competition working on the basis of this excitation will lead to the formation of the place fields (bottom-right). . . . . . . 35 xx LIST OF FIGURES 3.6 Illustration of the E%-MAX winner-take-all mechanism. (A) Architecture of the hippocampus network in de Almeida et al. (2009b, 2010). The granule cells in the dentate gryrus receives major convergent input from the entorhinal cortex. The pyramidal cells receive strong input from single granule cells and major convergent input from the entrohinal cortex. DG and CA3 exhibit E%MAX winner-take-all competition. (B) Competition is caused by the inhibitory interneurons that are capable of emitting global feedback inhibition (IPSP). (C) The interneurons are triggered after 3 ms of the first spike in the network cycle. Cells that are capable of reaching threshold during the 3 ms window also produce spikes, the other cells are inhibited before becoming active. Adapted from de Almeida et al. (2009a). . . . . . . . . . 36 4.1 Illustration of the process of place cell formation including LEC and MEC inputs. For each neuron, the excitement is computed for each position (x, y). Input from LEC is added to the input from MEC. At each specific position all cells compete through the E%-MAX process that outputs a population inhibition level. Rate map is build from the amount of excitation that exceeds the inhibition plane. This process is used in (RennóCosta et al., 2010a) and is analog to the one used in (de Almeida et al., 2009b, 2010). . . . . . . . . . . . . . . . . . . . . . . . . . 43 LIST OF FIGURES xxi 4.2 MEC and LEC inputs and estimation of model parameters. (a) Example of the 10 MEC modeled rate maps (number is the maximum firing rate). MEC rate maps remain constant during morphing. (b) Example of the 10 LEC rate maps from experimental data (H, maximum rate when informed, adapted from (Hargreaves et al., 2005) and 10 from the model for the two environments (square and round, maximum rate in both environments). Rate maps presented with equally distributed spatial score (ranked from right to left). (c) Histogram of spatial information score from LEC rate maps (H, experimental data and square, model. correlation 0.9957, P < 0.05). (d) Ratio (α) between the mean firing rates in MEC and LEC estimated as 0.32 by fitting to the experimentally observed reduction on spatial coincidence using population vector correlation as the environment is morphed (Leutgeb et al., 2007). (e) Histogram of the number of place fields found in DG cells (Leutgeb and square environment). Stable high correlation between experimental and simulated histograms during morphing indicates that modification in LEC activity do not disrupt place field formation (R = 0.98). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.3 Difference in spatial coincidence reduction rate for abrupt and linear interpolated morphing of LEC spatial response. Comparison of the mean population vector (PV) during remapping compared with (Leutgeb et al., 2007). For 17% of morphing, the minimal correlation value is 0.82 ± 0.01 compared to 0.75 observed experimentally. . . . . . . . . . . . . . . . . . . . . . . . . 50 4.4 Simulated DG cells exhibit independent place field rate remapping, as observed experimentally. Differential rate changes in individual firing fields of cells from the dentate gyrus during progressive maneuvering of the walls of the arena. (a) Recorded cells. Adapted from (Leutgeb et al., 2007). (b) Simulated cells. Individual fields are numerically labeled to relate to the respective line diagram of the mean field rate. The rate curves were fitted to linear (red), quadratic (green) or sigmoid (blue) functions and are shown when significant (p < 0.05, dotted line). (c) Histogram of the best fit classification for recorded and simulated curves. Correlation between histograms is of 0.9543 (P = 0.045). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 xxii LIST OF FIGURES 4.5 Different mechanisms for independent rate remapping of different place fields of the same cell. (A) Rate is directly affected by changes of the input drive. For a given cell, morphing (round to square) induces localized variation of LEC input, changing the rate of each place field independently (At PF1 , elevation of input drive (INPUT1 ) causes the rise of rate (RATE1 ). At PF2 , the fall of the input level (INPUT2 ) leads to reduction of rate (RATE2 )). In this case, remapping is only caused by the change of the input since the global inhibition level does not vary (dotted red line); (B) Rate is inversely affected by changes of the inhibition. Morphing induces localized variation of the global inhibition level, changing the rate of each place field independently (At PF1 , the raise of the global inhibition level (INH3 ) causes the decay of the rate (RATE3 ). At PF2 , the fall of the global inhibition level (INH4 ) causes the rise of the rate (RATE4 )). In this case, remapping is only caused by the local changes on the global inhibition level since all inputs to this cell remain in the same level during remapping. The change of the inhibition level is caused by variations of the input drive of the most excited cell. For each cell and wall shape a rate map is shown with the relevant place fields indicated by a white circle and the process values used of the computation of the rate at these place fields: the sum of entorhinal input (light gray bar for LEC and dark gray bar for MEC); the sum of entorhinal input of the most excited cell (red line); the global inhibition level (dotted red line) and the rate (black bar). . . . . . . . . . . . . . 53 4.6 Distribution of the mechanism balance ratio through active place fields. For clarification see Methods. Low ratio indicates prevalence of first mechanism (Figure 4.5A) while high ratio indicates that second mechanism (Figure 4.5B) is more effective. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 LIST OF FIGURES xxiii 4.7 DG cells do not represent sensory and spatial information in the same way as LEC cells. (A) Plot of the mean spatial information score on both shapes and the shape information score of LEC cells (blue, x) and DG cells (red, o). Spatial information score measures quantitatively how the position is encoded by one spike while shape information score relates to how much information about the shape each spike carries. (B) Box plot of the mean spatial information score and of (C) the shape information score for both populations. In each box, the central mark is the median, the edges of the box are the 25th and 75th percentiles and the whiskers extend to the most extreme data points. (D) The relative contribution of LEC and MEC input influences spatial properties. Histogram of the mean place field size as function of the ratio (α) of the mean drive of MEC and LEC onto EC. Low alpha indicates high LEC influence while low alpha indicates stronger MEC input. . . . . . . . . . . . . . . . . 70 4.8 Experimental variance correction for simulated data. (A) Mean population vector (PV) correlation for two successive recordings of the same morphing stage decays linearly with the increase of frequency proportional variance. The number of cells considered for the PV influences the effect of variance in correlation: less cells raises sensibility. To correct the simulated data to the experimental condition of (Leutgeb et al., 2007) we used the variance/frequency value (experimental factor β) that fits the experimental PV correlation. (B) Fitting of β is influenced by both number of cells and LEC/MEC mean rate ratio. . . . . . . 71 5.1 Rate remapping in the DG with spiking neurons. (A) Sample cells from MEC and LEC in the two environments. (B) Two sample DG cells exhibiting rate remapping. (C) PV correlation curve for the DG population compared with data from Leutgeb et al. (2007). . . . . . . . . . . . . . . . . . . . . . . . . 76 5.2 Distribution of the number of place fields with the spiking model. Distribution for both DG and CA3 (no recurrents) is coherent with experimental findings (Leutgeb et al., 2007). . . 78 5.3 Rate remapping in the CA3 population without recurrents don’t explain experimental data. PV correlation curve for of the simulated CA3 population aligned to experimental CA3 (red ) and DG (blue) curves (Leutgeb et al., 2007). . 79 xxiv 5.4 5.5 5.6 6.1 6.2 LIST OF FIGURES PV correlation curve for CA3 with recurrents batchtrained. Shown for multiple recurrent strengths (from black to light blue). Experimental curve, normalized to mean 1-1’ correlation in simulation, is shown in dotted red (Leutgeb et al., 2007). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Pattern completion by recurrent excitation. Trace of the potential of a neuron in a single gamma cycle for two different morphing stages. (top) When there is no recurrent input, the 10% morphing changes the input in a way that the cell cannot accumulate enough energy to release a spike. (bottom) When a recurrent input is present, the cell gets an extra amount of energy and spikes before the global inhibition is released. . . . . 81 Effect of LTP in the PV correlation curve. . . . . . . . . . 82 Experimental protocol. Multiple Y-maze (lef t) shown with its graph representation (right). Decision points represented as vertexes, spatial actions as straight arrows and nonspatial actions as angular arrows. All affordances are shown. Optimal solution is in red. (A) Spatial task. Reward is delivered when the agent reaches a specific location and applies a nonspatial action. Illustrated as the task in which the rat has to find and eat a piece of cheese. (B) Mixed spatial/nonspatial task. Rewarded action (∗) is only available at the goal location after the agent applies a nonspatial action at a different location (§). Illustrated as the task in which the rat has to pull a button to release water in the fountain located elsewhere. . . . . . . . . . . . . . . . . . . 87 The DAC architecture. (A) System overview and its major connections. The reactive layer relates statically sensory information and allostatic regulation with the motor primitives. The adaptive layer builds on top of the reactive layer with selforganized responsive units of perception, proprioception and actions. (B) Relevant connectivity in the medial entorhinal cortex. Dentate Gyrus (DG) and CA3 integrate the multimodal input from the lateral and media portions of the entorhinal cortex (LEC and MEC). Sequencing is obtained by the interconnectivity between CA3 and DG. Output is channeled back to the cortex through the CA1 and the Subiculum (SuB) (C) Schematic for behavioral learning (LTM acquisition) in the contextual layer. (D) Schematic for action recall. (E) Procedure to select next action. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 LIST OF FIGURES xxv 6.3 Spatial selectivity is sufficient for solving a spatial task. (A) Performance in the spatial task (optimal solution length 2.8±1.1 actions) as a function of the memory sequence length (median and interquartile range). (B) Estimated memory sequence length that leads to the best performance as a function of the mean length of the optimal solution of the maze. . . . . . 92 6.4 Conjunctive spatial/nonspatial selectivity is necessary for solving a mixed spatial/nonspatial task. (A) Percentage of successful trials of the spatial controller in the mixed task (optimal solution length 4.8±1.0 actions) and (B) performance relative to the shortest path in the mixed task as a function of memory sequence length (median and interquartile) for the halfshortest/-longest solutions. (C) Estimated memory sequence length that leads to the best performance as a function of of the mean length of the optimal solution of the maze. . . . . . . . 93 6.5 How conjunctive place cells solve the mixed spatial/nonspatial task. (A) 4-memory sequences of non-conjunctive place cells that cause misleading by attracting to action ∗ before action §being executed. The middle memory sequence is a special case in which the action §will never be accessible if the agent is at a locked position. (B) Conjunctive place cells solve it by establishing an independent graph for each behavioral context, causing that the agent will not be attracted to ∗ before executing §. . . . 95 6.6 Maze samples. With 10, 30, 100 and 129 decision points (topleft, top-right, bottom-left, bottom-right). Decision points in red and path in gray. . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.7 SF Robots. (left) Outdoor and (right) indoor units with 1.1 x 0.6 m and 0.6 x 0.6 m respectively. Both equipped with embedded computation, proximity sensors and color camera mounted on a pan-tilt unit. . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.8 Outdoors arena. (A) Overview of the operative area. (B) Snapshots of the robot (90o between each other). . . . . . . . . . 106 6.9 Neural representation of space. Rate maps of (A) three visual layer cells and (B) three memory layer cells. X- and Yaxis represent position and brightness the unit activity. (C) Population vector self-correlation for both layers. (D) Boxplot of spatial-info score in bits/spike. . . . . . . . . . . . . . . . . . . 109 xxvi LIST OF FIGURES 6.10 Sequence-learning task. (A) Robot goes forward until the red mark is detected. (B) Two color options are presented to the robot. (C) If the correct color is selected then two new options are presented. . . . . . . . . . . . . . . . . . . . . . . . . 111 6.11 Resource-localization task. (A) Experimental space (20 x 15 meters) with the arena marked on the floor. (B) Cue gathering at home. (C) Kidnap procedure. Robot is taken from one position to another and successfully finds the cued rewarded location. . . 112 A.1 Robot overview. The color camera is mounted over the pantilt unit in the front segment of the robot. The control CPU is placed over the middle segment. The load is placed over the rear segment. The proximity sensors are placed all around the robot, but only the ones in the front and in the back are used for the experiment. Robot produced by Robosoft. . . . . . . . . . . . . . 125 A.2 From face perception to gaze action: the perception/action reflex in the visuomotor loop (A) Illustration of the visual field of the robot and (B) the associated salience map emitted by the face detection and salience map system. (C) Through a competitive process the most salient point in the visual field is selected. If the most salient point is active in a zone associated with a gaze action, a saccade is activated. In this specific case, the action “Pan Left” is triggered, moving the detected face to the center of the visual field. (D) Illustration of the two possible action types ("tilt" and "pan”) in relation to the gaze direction. . 126 A.3 Visual loop organization. Perception excites action. Action inhibits the drive. The drive inhibits perception. Action and drive have spontaneous activity. Drives regulates the system by allowing action spontaneous activity to take place when few actions are triggered. . . . . . . . . . . . . . . . . . . . . . . . . . 128 A.4 Navigation loop organization. Action has spontaneous activity. Action can be inhibited by the perception and by the drive. The drive is activated by the action. . . . . . . . . . . . . 130 A.5 Demonstration venue (right) and the robot (top-left), chocolates and candies are placed in a plate located in the back part of the robot body. . . . . . . . . . . . . . . . . . . . . . . . . 132 LIST OF FIGURES xxvii A.6 Simulated navigation data. (A) Time evolution (exponential fit) for the percentage of the area covered by delivery stops for five different arena sizes. (B) Sample of robot trajectories and delivery spot spatial distributions at different time windows for the 5921 m2 square arena. . . . . . . . . . . . . . . . . . . . . . . 133 A.7 Demonstration of the gazing behavior by the visuomotor loop in a sequence with a moving subject. . . . . . . . . . 134 CHAPTER Introduction The mammalian brain is a complex structure organized into many components with different functionality and morphology. The hippocampus is one of these components. It is a subcortical structure located in the medial temporal lobe with abundant connectivity to many other cortical and subcortical regions. The anatomical preeminence of this brain region highlights its relevance for neuroscience and the study of the nervous system. In the last 60 years, strong research effort has been directed to the study of the hippocampus. In this context, the discovery of spatial selective neurons had the highest impact. Popularly known as “place cells”, these cells exhibit sustained and reliable increase in the rate of activity whenever the animal is situated in a specific and well-delimited area of the environment (O’Keefe & Dostrovsky, 1971). Empowered by its scientific charm and inspired by prior theories of spatial behavior in psychology (Tolman, 1948), the observations of place cells fostered a dominant theory that the main function of the hippocampus is to provide a cognitive map of the environment (O’Keefe & Nadel, 1978). Nowadays, with the 2 1 3 advent of many studies that were inspired by the "cognitive map" theory, there is plentiful experimental knowledge about how the "place cells" represent the space. Examples are the recognition of the effect of many environmental manipulations in the firing of place cells (Muller & Kubie, 1987; Fenton et al., 2000a); the observation of spatial selectivity in the first days of newborn rat pups (Langston et al., 2010); and the identification of place cells in the hippocampus of several species of mammals, from bats (Ulanovsky & Moss, 2007) to humans (Ekstrom et al., 2003). The "cognitive map" theory also promoted many theoretical and modeling studies that accompanied the experimental research and provided valuable insight in the understanding of experimental data. Nevertheless, the "cognitive map" is not the only theory of the hippocampus. It is also associated to the hippocampus a role in memory consolidation (Nadel & Moscovitch, 1997) and declarative memory formation (Cohen & Squire, 1980). The most influential evidence is that lesions on the hippocampus in humans cause severe retrograde amnesia that is not limited to spatial memory (Scoville & Milner, 1957). This famous clinical-study by Scoville called much attention and carried weight for the medial temporal lobe research field. In this direction, experimentation with humans revealed that some hippocampal neurons were selective to faces and objects (Fried et al., 1997) and to purely abstract concepts such as a character of a TV show (Quiroga et al., 2005). In rats, cells in the same region where place cells are recorded have shown selectivity to other physical aspects of behavior such as time (MacDonald et al., 2011). An experiment by Wood et al. (1999) identified hippocampal cells in rats that were selective to events rather than to the location where they took place in a recognition memory task. Altogether, these facts stress the hippocampus role in highly cognitive tasks that are not necessarily related to spatial cognition and behavior. 4 CHAPTER 1. INTRODUCTION Nowadays, the disjunction of hippocampus research is a major challenge in the field. It is not clear to what extend memory and spatial selectivity overlap. So far, these two lines of thought did not converge. It is unknown whether memory and spatial cognition functions share the same neural circuitry or whether there is a major functional dissociation in the hippocampus. Nadel, co-postulator of the cognition map theory, was aware of this kind of debate and positioned himself by labeling the hippocampus as “a spatial memory system” rather than a simply spatial system (Nadel, 1991). Nadel argued that the spatial aspect of memory was an “ineliminable property of our experience in the world” and therefore the hippocampus memory system could not be dissociated from spatial cognition. The viewpoint from Nadel is not consensual. Indeed, there is a crescent and already dominant view that space is one aspect (of many) of the memory represented by place cells (Eichenbaum, 2000). Evidence come from the observation that the hippocampus is also implicated in nonspatial relational learning in both rats and humans (Bunsey & Eichenbaum, 1996) and that place cells activity is influenced by behavior (Frank et al., 2000). Moreover, a study by Rolls et al. (2005) identified in rats different groups of cells that were coding for position, for objects and for objectposition combinations. These observations indicate the possibility that memory and spatial selectivity are indeed different interpretation of the same phenomenon. Within this context, the studies developed in this thesis sought the identification of what we call The Hippocampus Code, i.e. the set of computational principles underlying input-output transformation of neural activity in the hippocampus. Our axiom is that memory and space selectivity are related to the same computational process, they just differ in the nature of the observation and not on the process itself. We propose 5 that a structural and functional description of the hippocampus and its neural circuits is the key for the convergence of the theories of memory and space. To allow the definition of a generic computational principle it is essential to show that it applies equally for different aspects of the information, i.e. spatial and nonspatial data should be processed in the same way. There is however a major hitch in the analysis of this process. Most studies present the spatial and nonspatial aspects of the neural response in a way that they do not allow quantitative comparison. That’s mostly because of the special nature of space: whilst the "where" information is presented in a quasi-continuous and highly structured plane, the other aspects are in general presented in binary quantification (yes/no) and lack formal organization. This allow the distorted interpretation that space is more important than other aspects of memory or that nonspatial selectivity is built on top of the spatial representation, which constitutes an "annotated cognitive map". The unraveling of this problem was produced by an ingenious setup of Leutgeb et al. (2007). In this study, rats where placed in an arena whose walls could be gradually morphed from one shape to another. This experimental apparatus allowed a controlled, graded and structured parametrization of the nonspatial variable. The experiments with place cells revealed the "rate remapping" phenomenon in which place fields were kept stationary whilst their peak rate was gradually modulated by the morphing of the walls. This particular experiment allowed for the first time a quantitative analysis of the nonspatial aspects of memory with the same methods used for the place cells. Inspired in the experiment by Leutgeb et al. (2007), we investigated how the hippocampal representation is created with respect to both spatial 6 CHAPTER 1. INTRODUCTION and nonspatial aspects of memory (Chapter 4 and 5). For that, we modeled the mechanisms underlying nonspatial selectivity of place cells by quantitatively fitting to the spatial and nonspatial selectivity observed in the rate remapping process. Our computational study was based on a biologically-constrained model of how spatial selectivity emerges in the place cells if the known proprieties of its cortical inputs are considered (de Almeida et al., 2009a,b, 2010). By considering the nonspatial cortical input, we could demonstrate that the same neural mechanism underlies spatial and nonspatial selectivity in the dentate gyrus, the first stage of the hippocampus (Rennó-Costa et al., 2010a). Our results could quantitatively explain the experimental data (Leutgeb et al., 2007). In a second study, we based on the dynamics of the nonspatial selectivity observed with rate remapping to provide a novel interpretation on how attractor dynamics support pattern completion in the hippocampus. Moreover, our analysis allowed a quantitative analysis of the dynamics of plasticity in the formation of stable memories in the CA3 region of the hippocampus (Rennó-Costa et al., 2012a). With the computational model of the representation structure in place, we could study the function of the neural circuits (Chapter 6). We used the link between hippocampal activity and behavior identified in the biomimetic robotic-oriented cognitive architecture Distributed Adaptive Control (DAC) (Lisman, 2007; Verschure et al., 1992, 2003) to study if the mixed spatial and nonspatial representation is essential for the ability to solve problems in mixed spatial/nonspatial tasks (Rennó-Costa & Verschure, 2012). For this reason, we used a virtual and mathematically defined action-space that allowed a quantitative analysis of performance of the hippocampal-based controllers. In a further step, we investigated whether the same principles hold in real-world environments. For that, we implemented the DAC architecture in an unmanned mobile vehicle and tested it in spatial, nonspatial and mixed spatial/nonspatial tasks 7 (Rennó-Costa et al., 2011). Altogether, the experiments included in this thesis provided valuable insights about The Hippocampus Code (Discussions in chapter 7). Our major contribution has been the demonstration that spatial and nonspatial information are processed through the same mechanisms and that the conjunctive representation is essential for real-world cognition and behavior. This allowed an interpretation in which memories and place selectivity are indeed instances of the same computational concept. Completing this thesis, we have in the next two chapters a non-exhaustive review of the current knowledge of the hippocampus (Chapter 2) and the available computational models of the medial temporal lobe (Chapter 3). CHAPTER The hippocampus The hippocampus is located in the medial temporal lobe along with the parahippocampal gyrus, which include the perirhinal, parahippocampal and entorhinal cortices (Figure 2.1). It is elongated1 through the dorsoventral axis with the synaptic junctions distributed alongside the dorsoventral and the mediolateral axis. The entorhinal cortex is the major synaptic interface from the neocortex to the hippocampus by the projection of its superficial layers to all hippocampal parts and by receiving projections in its deep layers mainly through the subiculum. The medial temporal lobe connects to mostly all brain regions (Bird & Burgess, 2008), from the specialized areas of the neocortex such as the visual cortex and the prefrontal cortex (Degenetais, 2003) to subcortical areas such as the amygdala (Pitkänen et al., 2000) and the ventral striatum (Pennartz et al., 2011). The hippocampus formation is subdivided in several regions (Figure 2.2): the dentate gyrus (DG), the cornu ammonis (CA) areas (CA1, CA2, CA3 1 Hippocampus is the latin name of a sea monster from Greek Mythology whose elongated outline reassembles the shape of the hippocampus in the brain. Hence the name used in biology. 8 2 9 Figure 2.1: The medial temporal lobe in the brain. (top) Coronal cut of whole-brain Macaca mulatta (bottom) Nissl-sagittal cut of whole-brain Rattus Norvegicus. Highlighted areas: dentate gyrus (DG), CA1, CA3, subiculum (Sub), entorhinal cortex (EC, lateral LEC, medial MEC), and perirhinal cortex (36). Adapted from brainmaps.org. 10 CHAPTER 2. THE HIPPOCAMPUS Figure 2.2: Hippocampus anatomy. Drawing of the neural circuitry of a rodent hippocampus by Ramón y Cajal (1909). Diagram with most relevant connections within the hippocampus and the entorhinal cortex. and CA4) and the subiculum (Sub). An extensive review about the hippocampus wiring is provided by Johnston & Amaral (1998). The dentate gyrus is composed mainly by granule cells and by inhibitory interneurons. The perforant pathway (PP), originated in the entorhinal cortex, and the mossy fibers, originated in the mossy cells that receive input from the CA3, PP and other granule cells, provides major innervation to this area. In the CA3, pyramidal cells receive input from the PP and DG and other CA3 pyramidal cells. It is remarkable that the information flow is not purely sequential but rather exhibits cyclical processing with the DGCA3 links (Lisman, 1999). In the CA1, pyramidal cells receive input from the PP and CA3 and project back to the entorhinal cortex through the Sub. Most of the synaptic organization in the hippocampus is conserved across species (Manns & Eichenbaum, 2006). Another salient characteristic of the hippocampus is that it is the locus of the first observation of Long-Term Potentiation (LTP) (Lø mo, 1966), a special kind of synaptic plasticity in which the signal transmis- 11 sion between two neurons is augmented following synchronous activation (Bliss & Lø mo, 1973). The anatomical properties of the CA3 region – in special the interconnectivity between the pyramidal cells – in addition to the observation of LTP have inspired models of recurrent auto-associative neural networks, in special the Hopfield network (Hopfield, 1982). The parahippocampal gyrus is organized in a way that the entorhinal cortex is the main interface of the neocortex with the hippocampus (Witter et al., 2000a). The entorhinal cortex has a major anatomical dissociation between its lateral and medial regions (Witter et al., 2000b). The two parts are also distinguishable in the connectivity, having the lateral entorhinal cortex (LEC) connected favorably with sensory driven areas such as the olfactory, insular and perirhinal cortices while the medial entorhinal cortex (MEC) is mainly connected with visual-spatial occipital, parietal and postrhinal cortices and the pre-parasubiculum (Canto et al., 2008). There is no noticeable difference in how LEC and MEC project to the dentate gyrus in rats while some topological organization can be observed in monkeys (Witter et al., 1989). One important aspect of the hippocampus morphology is that it remains fairly stable through many species, from rodents to primates. This evolutionary constancy evidences the fundamental role of the hippocampus in the brain on the support of cognition and behavior. The implication of this observation is that conclusions from animal experimentation can in many cases be very influential on the study of the human brain. Regarding its function, the medial temporal lobe is mainly associated with three purposes: inhibition, memory and space. The inhibition function is related with the fact that animals with hippocampal lesions exhibit motor hyperactivity, which allowed the link to anxiety disorder (Gray & McNaughton, 2000). 12 CHAPTER 2. THE HIPPOCAMPUS The first link between memory and the hippocampus was originated on the reports from the clinical studies of the patient Henry Gustav Molaison (H. M.) by Scoville & Milner (1957). H. M. had most of his medial temporal lobe surgically removed as a treatment to epilepsy, which resulted in partial retrograde amnesia - the patient could remember memories from many years before the surgery but was not able to recall facts and events that happened a few years before the surgery – and severe anterograde amnesia – the patient could not remember events that have just happened. This outcome reveled an important role of the medial temporal lobe in the formation of new memories. Following studies have related the hippocampus and the medial temporal lobe with declarative and episodic memory functions, with the specific functions of memory consolidation, relational cognition and the link between memory and space (Eichenbaum, 2001). The parahippocampal gyrus also has some specific functionality regarding memory cognition. For instance, the entorhinal cortex is associated with working memory of novel objects (Hasselmo & Stern, 2006) while the perirhinal cortex is related with familiarity-based object recognition (Murray et al., 2007). The latest function attributed to the hippocampus is spatial cognition. The major finding supporting this functionality was the observation of the place cells by O’Keefe & Dostrovsky (1971) in the CA1 region. Place cells were also found in the dentate gyrus and CA3 (Jung & McNaughton, 1993; Leutgeb et al., 2007) and in the subiculum (Brotons-Mas et al., 2010; Sharp, 2006). Regarding the parahippocampal gyrus, spatial selectivity is also observed in the medial entorhinal cortex (Fyhn et al., 2004) but not in the lateral entorhinal cortex (Hargreaves et al., 2005). Evidence that place cells is effectively associated with behavior is the fact that performance in spatial tasks is impaired after hippocampal damage in both rats (Morris et al., 1982) and humans (Astur et al., 2002). More- 13 over, patients with hippocampal damage have impaired spatial memory recall (Bohbot et al., 1998; Bartsch et al., 2010). While the three cognitive functions associated with the hippocampus might seem uncorrelated, there is evidence for functional overlapping and neural circuitry sharing between them. For instance, place cells firing is modulated by changes in nonspatial features of the environment (O’Keefe & Conway, 1978) and by behavioral context, which can be related to episodic memory encoding (Wood et al., 2000). Moreover, inhibition might affect spatial cognition as a process of attention with implications in place cells properties (Fenton et al., 2010). However, these finding does not overrule neural circuitry specialization, as there is evidence for functional dissociation in the septotemporal axis of the hippocampus in spatial tasks (Bannerman et al., 1999) and inhibitory learning tasks (McDonald et al., 2006). The available evidence also delimits the functional boundaries of processes in which the hippocampus and the medial temporal role are not involved or, at least, do not play a fundamental role. One example comes from the observations that H. M. was capable of learning new motor skills such as drawing (Corkin, 2002), rotary pursuit, bimanual tracking, and tapping (Corkin, 1968) and the ability to learn certain problem-solving procedures (Cohen & Corkin, 1981). The same effect is also observed in rat in regards to spatial cognition (Morris et al., 1982). In the Morris water maze, rats with hippocampal lesion are initially impaired of finding the location of a hidden platform. However, after a long period of learning they succeed in accomplish the task. Indeed, H. M. “was able to construct a cognitive map of the spatial layout of his house as the result of daily locomotion from room to room” (Corkin, 2002). These are evidence of dissociation between procedural non-conscious memory, whose learning is not dependent on the medial temporal lobe, and declar- 14 CHAPTER 2. THE HIPPOCAMPUS ative memory, which is dependent on the medial temporal lobe (Cohen & Squire, 1980). 2.1 Place cells The recording of place cells was made possible by the development of single-unit (one neuron) recording technics. The technological breakthrough was the invention of immovable implantable electrodes that could reach specific brain areas and its further evolution to stereotrodes (McNaughton et al., 1983) and tetrodes (Recce & O’Keefe, 1989). Singleunit spikes trains can be identified in a multi-unit recording by their differential amplitude and waveform. From the same recording is possible to obtain the local field potential (LFP), which is the electrical current flowing within a certain volume of tissue and that, thus, includes the activity of multi-units. The identification of the place cells followed the simultaneous recording of the activity of single neurons and the position of the animal (Figure 2.3A). By plotting the spikes of a neuron on top of the trajectory of the rat (Figure 2.3B), O’Keefe & Dostrovsky (1971) observed that some CA1 neurons were only producing spikes at a specific region of the arena. This discovery was highly influential in neuroscience and after many years much is known about the spatial properties of the hippocampal neurons. Place cells were also found in the dentate gyrus and CA3 (Jung & McNaughton, 1993; Leutgeb et al., 2007) and in the subiculum (Brotons-Mas et al., 2010; Sharp, 2006). Moreover, place cells were also found in the hippocampus of humans (Ekstrom et al., 2003), monkeys (Hori et al., 2005), bats (Tsoar et al., 2011) and birds (Hough & Bingman, 2004). 2.1. PLACE CELLS A 15 B CAMERA ARENA Figure 2.3: Illustration of the methods for the recording of place cells. (A) Recording setup. Rat with implanted immutable drive with multiple tetrodes is free to move inside an arena. A camera is used to track the position of the rat. Tracking and neurophysiological data are time stamped. (B) Common place cell representations. (top) Spikes (red dots) overlying the trajectory of the rat during one recording session. (down) Rate map of the spatial activity. Frequency varies from highest (red) to silent (blue). To better understand what is the nature of the place cell representation, many studies have used environmental manipulations as a study paradigm. Muller & Kubie (1987) found that rescaling and rotating a circular arena evoked similar transformations in the rate maps of the place cells (Figure 2.4AB). They could predict the activity of the hippocampal cells based on the transformations. They also observed that some cells were linked to distal cues (the walls of the lab where the arena was placed) while some other cells were linked to proximal cues (a white card placed inside of the arena). Also, they found that place cells do not show direction selectivity in open field experiment but they exhibit strong polarization in linear tracks in maze experiments (Muller et al., 16 CHAPTER 2. THE HIPPOCAMPUS 1994). Their major finding however was that place cells remap. When the rat is placed in a different arena – even though it has the same characteristic than the original – the activity of the place cells cannot be predicted from the previous experiment (Figure 2.4C). In other words, the place fields are morphed to a random position. Moreover, they observed that many cells are active in both environments but some are only active at one of the environments. When returned to the original environment the place cells returned to their initial firing profile. These findings allow an interpretation that place cells have not only a general and relative spatial representation of the space but it spatial representation is related to specific locations. Another interesting observation is that place cells can still exist when the rat is in the dark (Figure 2.4D), although with a lower reliability and in minor number (Markus et al., 1994). Indeed, the firing of hippocampal place cells in the dark depends on the rat’s recent experience (Quirk et al., 1990), which might suggest some kind of spatial memory recall. These results point out that the rat is capable of building a fairly reliable representation of the space when deprived from sensory stimuli. Together with the fact that such representation depends on previous anchoring on environmental cues, the evidences suggest the existence of some kind of relative path integration system underlying the formation of the place cells. This could be clearly observed in an ingenious study by Gothard et al. (1996) in which the initial position of a linear track could be manipulated. Place cells were dominantly driven by the environmental cues when these were available (light condition) since the place fields were stable in relation to the endpoint of the linear track regardless of the initial position. However, when sensory deprived (dark condition) the place fields were stable in relation to the initial position and not to the 2.1. PLACE CELLS 17 A C B D Figure 2.4: Illustration of the spatial proprieties of place cells. (A) Rotation of dominant environmental cues (such as a white card) causes similar transformation in the place cells. (B) Changes in the dimensions of the environment causes similar transformation in the place cells. (C) Different arenas (represented with black and gray border) with the same dimensions cause place cells to exhibit uncorrelated spatial activity. (D) Place cells are stable in the dark if the rat had experienced the environment with light previously. endpoint, suggesting that the rat was somehow “counting steps”. It is also important to mention the time properties of the spike train of the place cells. The hippocampus activity is strongly modulated in two frequency ranges: theta (∼ 8 Hz) (Green & Arduini, 1954) and gamma (∼ 40 Hz) (Soltesz & Deschênes, 1993). The time of a spike of hippocampal cells tend to be confined to a specific phase of the gamma cycle (Bragin et al., 1995), which suggests that the mechanism that evokes the spike is also responsible for this oscillation (de Almeida et al., 2009a). Regarding the theta rhythm, it is present during motor activation and REM sleep (Vanderwolf, 1969). The most relevant propriety regarding 18 CHAPTER 2. THE HIPPOCAMPUS spatial activity is a phenomenon named phase precession: the position of the rat inside of the place field (how close it is to the center) can be predicted from the phase of the firing within the theta cycle (O’Keefe & Recce, 1993). The relation of theta and gamma rhythm is of major relevance for the understanding of the functionality of the neural circuitry in the hippocampus. For instance, it allows the organization of time-compressed discrete sequences which might be related to trajectories (Skaggs et al., 1996) or even as a forward path for a future trajectory (Johnson & Redish, 2007). The observation of replays of sequences during sleep is an indication of memory consolidation in the hippocampus (Girardeau & Zugaro, 2011). The organization in sequences seems to be a computational principle of how the hippocampus deal with memory (Jensen & Lisman, 1996) and plays a major role in how it affects behavior (Lisman, 2007). 2.2 Other spatially driven cells A B C 90º 180º 0º 270º Figure 2.5: Illustration of other spatially selective cells. (A) Rate map of a grid cell with triangular grid organization. (B) Polar plot of the angular response of a head direction cell. (C) Rate map of a border cell in an arena with an internal wall. Place cells are not the only spatially driven cells in the brain. The same technic used to identify place cells when applied to other brain regions allowed the identification of additional spatial selective cell types such 2.2. OTHER SPATIALLY DRIVEN CELLS 19 as the grid cells (Figure 2.5A) in the medial entorhinal cortex (Fyhn et al., 2004; Hafting et al., 2005), the head-direction cells (Figure 2.5B) found in the post-subiculum (Taube et al., 1990b,a) and the border cells (Figure 2.5C) also found in the medial entorhinal cortex and also in the parasubiculum (Solstad et al., 2008). The grid cells exhibit multiple place fields organized in a triangular grid that spans throughout the whole environment (Figure 2.5A). The spatial pattern of grid cells can vary the spatial phase, angular phase and scale (Figure 2.6) (Hafting et al., 2005). Moreover, cells located in the same micro-region exhibit the same angular phase and scale but not the same spatial phase, suggesting a network property. Grid cells are topographically organized in the dorsoventral axis of the medial entorhinal cortex having the intervertex distance varying from 25 cm to 8 meters (Brun et al., 2008b). The scale span is not continuous but discrete (Barry et al., 2007), which supports the anatomical evidence of the existence of multiples network cores (Witter & Moser, 2006). Grid cells are found on the superficial layers II and III and in the deep layer V and the parasubiculum (Sargolini et al., 2006). In the layer III there can be found conjunctive grid cells that the response is also selective to orientation as in the case of head-direction cells. This propriety allow the indirect observation of grid cells in humans using fMRI data (Doeller et al., 2010). Grid cells are believed to be the main mechanism of path integration in the medial temporal lobe (McNaughton et al., 2006). Major evidence come from the fact that the grid cells remain stable whenever CA3 cells undergo rate remapping (Fyhn et al., 2007). There are two kinds of models that explain the grid cell formation: one based on continuous attractor networks (Samsonovich & McNaughton, 1997; Guanella & Verschure, 2006) and the second based on oscillatory interference (Burgess et al., 2007; Hasselmo et al., 2007). Both models agree in the fact that position 20 CHAPTER 2. THE HIPPOCAMPUS A B C Figure 2.6: Illustration of the spatial variations of grid cells. (A) Scale or intervertex distance: the distance between place fields in the same triangular formation. (B) Angular phase: the angle between place fields in the same triangular formation. (C) Spatial phase: the absolute position of a specific place field in the x- and y-axis. is integrated based on odometry values such as speed and head-direction. Although there is evidence for independence of the head direction and grid cells systems (Whitlock & Derdikman, 2012), the fact that they are sensible to the same kind of environmental manipulations (Taube et al., 1990b) suggests that the head direction system is situated upstream, affecting the grid formation. These models are reviewed in the Section 3.1.1. Path integration in rats is computed on the basis of purely internal signals, such as vestibular or proprioceptive afferences (Etienne et al., 1998). However, visual cues are also is used to calibrate the path integration system to known landmarks. Evidence come from one experiment in which 2.3. CONJUNCTIVE PLACE CELLS AND RATE REMAPPING 21 rats were trained to explore and return to the nest Etienne et al. (2004). In both dark and light conditions the rats were able to perform. However, when the position of the nest (and the visual landmarks) were rotated, the rats returned to the original nest position in the dark condition and to the adapted nest position in the light condition. The border cell is the latest kind of spatial selectivity observed in the medial temporal lobe. Not much is known about its implications in the formation of grid cells, although it is diversely distributed along the entorhinal cortex, along with the grid cells (Solstad et al., 2008). Moreover, they can have strong implication in the formation of place cells as anticipated by Hartley et al. (2000). 2.3 Conjunctive place cells and rate remapping There are many evidences that place cells are not only spatially driven. For instance, Wiener et al. (1995) have shown that some place cells were not fixed to the absolute position in an arena but to task specific locations. This was found by rotating the whole arena but the task specific places. Further work by Deadwyler et al. (1996) have shown that many CA1 and CA3 cells were coding for distinct task stages in a spatial delayed-nonmatch-to-sample task. These evidences, reviewed by Muller (1996), suggest that the location of firing is defined by other aspects and not purely by the spatial representation. The observation of rate remapping by Leutgeb et al. (2005) revealed that place cells are conjointly selective to spatial and nonspatial information. After making subtle changes in the environment, the researchers observed that the place cells kept stable place field location but exhibited different place field peak rate. The authors suggest that this phenomenon might 22 CHAPTER 2. THE HIPPOCAMPUS A B Rate 12 10 8 6 4 2 0 C E 1.0 PV correlation D Population Vector (PV) Rate 12 10 8 6 4 2 0 CA3 0.8 DG 0.6 0.4 0.2 Morph progression Figure 2.7: Illustration of the rate remapping phenomenon. (A) Different arena shapes that provoke gradual rate remapping and illustrative rate maps for CA3 (B) and DG (C). Red refers to high rate while deep blue refers to silent. Graph at right is the mean fire rate of the place fields according to the morphing phase. For DG, upper place field (black) and bottom place field (gray) are represented. (D) Procedure to compute the population vector (PV). The rate of all cells at a specific position are included in sequence in an array. The activity ensemble change is computed by the mean correlation of the PV of two conditions computed for every position. (E) Illustration of the PV correlation curve for DG and CA3. CA3 present no difference between 1-step morphing and two successive recordings of the same arena. Adapted from (Leutgeb et al., 2007). be the cause of single-cell remapping in task-specific, direction-specific and trajectory-specific changes in population activity. Further work by the same group (Leutgeb et al., 2007) has shown that the change in rate in an ensemble level is coherent with the gradual change in the environment (Figure 2.7). This shows that place cells are not only coding for nonspatial information but that they also do it in a gradual 2.3. CONJUNCTIVE PLACE CELLS AND RATE REMAPPING 23 fashion, as the coding for space. Moreover, the change profile of each place field is independent signifying that nonspatial selectivity is place and cell specific. CHAPTER Computational models of the medial temporal lobe In this chapter we will review (not exhaustively) the most relevant models of the medial temporal lobe. In this scope we include models of the hippocampus, the entorhinal cortex and the interaction between them. Most of these models are limited to one of the theoretical streams: spatial selectivity and memory. Although it is not rare to observe some overlapping. 3.1 Computational models of spatial selectivity Following the discovery of the place cells, there have been many attempts to model the formation of spatial selectivity in the hippocampus: from robotic-based systems that use extensive sensory information, path integration and reinforcement learning to theoretical models based on properties of cortical activity. With the discovery of the grid cells, the models started to link its activity to the place cells which also fostered the cre24 3 3.1. COMPUTATIONAL MODELS OF SPATIAL SELECTIVITY 25 ation of grid cells models. 3.1.1 Computational models of grid cells There are two main groups of computational models of grid cells: the oscillatory-interference models (Burgess et al., 2007; Hasselmo, 2008) predict that the hexagonal pattern emerges in single cells from the interaction of multiple oscillatory inputs; the continuous dynamics models (McNaughton et al., 2006; Guanella & Verschure, 2006) predict that the grid patterns emerge from network interaction. These models were extensively reviewed by Giocomo et al. (2011) and more recently by Zilli (2012). Oscillatory interference model The oscillatory interference model is a work by Burgess et al. (2007). It predicts that the grid pattern is a cellular feature. The model is based on an oscillatory interference phenomena in which the sum of activity of high-frequency oscillators creates a low-frequency pattern. The model predicts that the activity of the grid cell is a product of the synchronization of the intrinsic oscillation of three dendritic subunits (Figure 3.1). The frequency of each dendritic subunit is modulated periodically by the integral of the speed in a preferred direction. Speed information is available through the head direction cells (O’Keefe et al., 1998). When the frequency of the three dendritic subunits is in a high-state it give rise to the interference phenomena at the theta frequency, generating the place fields. In this model, the inter-vertex distance is dependent on the strength of the modulation of the oscillatory frequency by the speed of the animal. This variable can be dependent on cellular proprieties such as synaptic CHAPTER 3. COMPUTATIONAL MODELS OF THE MEDIAL TEMPORAL LOBE 26 A HD CELL 0º HD CELL 60º Dendritic subunit GRID CELL HD CELL 120º B Dendritic subunit 0º Dendritic subunit 60º Dendritic subunit120º GRID CELL Figure 3.1: Illustration of the oscillatory interference model. (A) Each grid cell has three dendritic subunits that receives input from different head-direction cells with phase intervals of 60. The dendritic subunits have an intrinsic internal oscillation that is modulated by the integral of the speed input of the head-direction cells. (B) A grid cells exhibit theta based activity when the three sub-units have similar frequencies that causes an oscillatory interference. The movement of the animal changes the frequency of each dendritic subunit leading to successive active and silent states that give rise to the grid pattern. weight that can vary dependent on the region of the medial entorhinal cortex. Angular offset is determined by the preferred orientation of the head-direction cells. Position offset can be set by a feedback phase-reset signal from the downstream place cells. 3.1. COMPUTATIONAL MODELS OF SPATIAL SELECTIVITY 27 Intrinsic persistent spiking model The intrinsic persistent spiking model is a work of Hasselmo (2008) and is a slight variation of the oscillatory interference model (Burgess et al., 2007). In this model the oscillation is produced by the head-direction cells persistent firing and not on the sub-threshold firing of the entorhinal cells. The bases on the idea that the activity of the grid cells is determined by the synchronicity of the head-direction cells upstream (Figure 3.2). The considered head-direction cells exhibit intrinsic persistent activity with fixed and uniform frequency. The grid pattern rise from the fact that the phase of each head-direction cell is modulated by the projection of the speed of the animal in the preference angle. This speed input does not change the intrinsic frequency oscillation but only the firing phase. The total phase shift is proportional to the integral of the projected speed. The grid cells receive input of three different head direction cells with phase intervals of 60 which allow the two-dimensional periodic pattern. In this model, the inter-vertex distance is dependent on the strength by which the oscillatory phase in the head-direction cells is modulated by the speed of the rat. As in the oscillatory interference model, this variable can be dependent on cellular proprieties that can vary dependent on the region of the medial entorhinal cortex. Angular offset is determined by the preferred orientation of the head-direction cells. Position offset can be set by a feedback phase-reset signal from the downstream place cells. Continuous attractor dynamic models The second class of grid cells models is based on continuous attractor dynamics. These models predict that the grid pattern is a network feature. The principle was created by Samsonovich & McNaughton (1997) and extended by McNaughton et al. (2006) to add spatial periodicity and by CHAPTER 3. COMPUTATIONAL MODELS OF THE MEDIAL TEMPORAL LOBE 28 A HD CELL 0º HD CELL 60º GRID CELL HD CELL 120º B HD CELL 0º HD CELL 60º HD CELL 120º GRID CELL Figure 3.2: Illustration of the intrinsic persistent spiking model. (A) Each grid cell receives input of three different head direction cells with phase intervals of 60. (B) A grid cells is active when the persistent firing of three head direction cells are synchronized. The movement of the animal changes the phase of each head direction cell leading to successive synchronization and dis-synchronization states that give rise to the grid pattern. Guanella & Verschure (2006) to allow the hexagonal grid pattern. The model consists of a neural network in which the neurons are organized in a bidimensional topology (Figure 3.3AB). Each neuron is connected to its closest neighbors through an excitatory connection and to distal neighbors through an inhibitory connection (Figure 3.3C). With this setup the system always converges into a state in which a stable bump of energy if formed. The bump of energy can be maneuvered towards a certain direction if the weights of the neighborhood connections are modulated at this direction (Figure 3.3C). By linking the speed and 3.1. COMPUTATIONAL MODELS OF SPATIAL SELECTIVITY A B C D 29 Figure 3.3: Illustration of the continuous attractor dynamic models. Bump of activity (black with high activity and white with low activity) in the bidimensional topological organization of the entorhinal cortex neural network with (A) rectangular neighborhood (McNaughton et al., 2006) and (B) triangular neighborhood (Guanella & Verschure, 2006). In both models the cells in the boundaries are interconnected allowing the formation of periodic place fields. (C) Bump is maintained by homogeneous lateral connectivity (shown in a linear representation for clarity). (D) Bump of activity is moved by the modulation of the lateral connectivity in the direction of the animal motion. the direction of the movement to the modulation of the weights it is possible to integrate the path by mapping the position of the animal to the position of the bump. The hexagonal grid pattern is obtained by the wrapping of the neural network. If the bump reaches a boundary, it appears in the other side of 30 CHAPTER 3. COMPUTATIONAL MODELS OF THE MEDIAL TEMPORAL LOBE the network. The organization as a twisted torus proposed by Guanella & Verschure (2006) allowed the formation of the triangular tessellation. The inter-vertex distance can be linked to the level of modulation of the weights by the speed vector. Every cell in the same population share the same inter-vertex distance and angular phase, what confirms experimental observation (Sargolini et al., 2006). The fact that the spatial frequency of the grid cells changes along the dorsoventral axis of the medial entorhinal cortex (Hafting et al., 2005) can be explained by the existence of multiple networks with different gains. Further evidence comes from the fact that the distribution of spatial frequencies is not continuous (Barry et al., 2007). Regarding the spatial phase, the bidimensional organization makes that each cell have a different spatial phase and that the set of cells is able to cover the whole phase-space in an uniform fashion. Another interesting point is that the bump can be set to a specific position by specific excitatory input patterns. Such patterns can be learned by hebbian rules when interacting with a place cell population. With this mechanisms the grid cells can be calibrated by the place cells, allowing the reset of the integrative error. Mixed cellular-network model A recent model (Navratilova et al., 2011) was able to merge the cellular features of the oscillatory-interference models with the network features of the continuous attractor dynamics model. In its model, a population of conjunctive grid cells (grid cells whose firing has a preferred angle like the head-direction cells) is build implementing cellular cellular mechanisms for grid formation. The bump in the continuous attractor network 3.1. COMPUTATIONAL MODELS OF SPATIAL SELECTIVITY 31 is driven by the connections of the population of conjunctive grid cells. This solves the issue of how the weights could be modulated by speed and also allow the emergence of phase precession. A similar approach is provided by Hasselmo & Brandon (2012). 3.1.2 Computational models of place cells Through time, many groups tried to model the spatial selectivity of the place cells. Most of these attempts followed two approaches: some use mathematical modeling to try to predict the place cell activity from known features of the cortical inputs to the hippocampus; others had a more technological approach using robots, cameras or graphical simulations of three-dimensional environments to emerge spatial selectivity from visual input. Some models also produce a synergy between the two approaches by studying how the two inputs interacts. There is still a third class of models that is not concerned about the formation of place cells but deals with its implication in behavior. Most of the models come with navigation strategies that use the cognitive map to plan the action sequence. Place cells from border cells A model of place cells based on the activity of border cells was proposed by Hartley et al. (2000) from the group of Neil Burgess. Interestingly, this model was published before the discovery of the border cells (Solstad et al., 2008), which was one of the predicitions of their model. They defined a special kind of cell named "boundary vector cell" (BVC) whose firing is selective to the borders in a specific direction (Figure 3.4A). In the text they even predicted that the border cells would be find in the 32 CHAPTER 3. COMPUTATIONAL MODELS OF THE MEDIAL TEMPORAL LOBE Figure 3.4: Illustration of the place cell model based on border cells. (A) Example of the typical rate map of a BVC (boundary vector cell) in three environments. (B) The activity of a place cells is obtained by the integration (followed by a linear threshold filter) of multiples BVC cells. entorhinal cortex. To build the place cell activity from the BVCs, each hippocampal cell receives input from multiple BVCs and apply a linear threshold filter (Figure 3.4B). The model could make interesting predictions regarding the population of place cells, including the transformations in the rate maps caused by the inclusion of barriers and by geometrical manipula- 3.1. COMPUTATIONAL MODELS OF SPATIAL SELECTIVITY 33 tions of the walls. In a latter work they also included plasticity in the connection between the BVCs and the place cells (Barry & Burgess, 2007). Plasticity allowed the reproduction of some experimental findings such as: the slow disappearance of duplicate place fields produced when a barrier is placed into a familiar environment (Barry et al., 2006); the fact that place cells tend to over-represent barriers placed inside of the arena (Rivard et al., 2004); and the parametric prediction on the change of the place field distribution after non coherent rotation of multiple visual cues (Fenton et al., 2000a,b). Place cells from grid cells A major breakthrough on the place cells models was the discovery of the grid cells (Hafting et al., 2005). That’s because the major input to the hippocampus is the innervation from the entorhinal cortex and, therefore, the activity of its cells is likely to be essential to place cell formation. Indeed, place cells are less stable after EC lesions (Van Cauter et al., 2008) and CA1 place fields are more sparse after MEC layer III lesions (Brun et al., 2008a). All models share the same principles that each place cell integrates input from multiple grid cells. The models differ in the strategy by which the activity at single place fields is isolated, which might include different learning rules and network inhibition. A first model by Solstad et al. (2006) identified a phenomena in which the overlap of multiple grid cells with different configurations leaded to an irregular rate map landscape in which single place fields could be observed. Based on this principle, the place cells could be formed by the sum of multiple grid cells followed by a linear threshold filter. The drawback of this model is that the difference between the peaks is highly 34 CHAPTER 3. COMPUTATIONAL MODELS OF THE MEDIAL TEMPORAL LOBE variable, making it hard to settle a threshold value with good overall fitness. This effect can be reduced by an efficient selection of synaptic projections, which can be accomplished by means of competitive learning (Rolls et al., 2006; Si & Treves, 2009). This effect is also present on an integrate and fire model of place cells with hebbian-learning and competition (Savelli & Knierim, 2010). The phenomena observed by Solstad et al. (2006) was latter formalized as part of a moiré interference effect (Blair et al., 2007). It was show the interference between the grid cells can cause the formation of place fields of a multiple scales and rotations. Another interesting group of models is provided by Jeffery (2011). To explain remapping on the place cells they assume that different grid cell patterns are created for each dendritic subunit of the hippocampal cells. By means of contextual gating it is possible that different grid cells drive the activity of the place cell at different contexts. Integrative-competitive model In their recent work, de Almeida and colleagues have identified two mechanisms underlying place cell emergence from grid cells in the dentate gyrus (de Almeida et al., 2009b) and CA3 (de Almeida et al., 2010): (1) the integration of multiple inputs with specific synaptic weight distribution (Figure 3.5 top) and (2) a competitive process based on a networkwide feedback inhibition named E%-MAX winner-take-all (de Almeida et al., 2009a) (Figure 3.5 bottom and 3.6A). The competition is ruled by a network on inhibitory interneurons that is activated by the hippocampus neurons (Figure 3.6B). Whenever the first spike is detected, the interneuron network inhibits the hippocampal neurons after a short activation delay (Figure 3.6C). This latency gives enough time for other 3.1. COMPUTATIONAL MODELS OF SPATIAL SELECTIVITY + = + 35 + E%-MAX WTA Figure 3.5: Illustration of how place cells are generated from grid cells in the integrative-competitive model. (top) The integration of multiple grid cells will generate a spatial dependent excitation for a granule cell (bottom-left) . The E%-MAX winner-take-all competition working on the basis of this excitation will lead to the formation of the place fields (bottom-right). neurons with high accumulated excitation to also fire. The E%-MAX principle defines an energy bottom-limit (measured as the maximum energy minus 10% (de Almeida et al., 2009a)) that separates the cells that will be active from the cells that will not. The translation into rate is due the dynamic nature of the neurons that forces the competitive cycle to run in a defined frequency in the range of gamma. By assuming a variable initial condition for each cell in each cycle is possible to assume a gradual mapping between the excitation and rate in the range above the bottom-limit. This mapping supports the emergence of the place field (de Almeida et al., 2009b). This approach is follow, at some point, the same integrative principle of Solstad et al. (2006). The major difference is that the E%-MAX mecha- CHAPTER 3. COMPUTATIONAL MODELS OF THE MEDIAL TEMPORAL LOBE 36 A CA3 E%-MAX WTA B pyramidal/granule cells - Dentate Gyrus + E%-MAX WTA inhibitory interneuron 3ms C Entorhinal Cortex potential cell 1 cell 2 cell 3 IPSP Figure 3.6: Illustration of the E%-MAX winner-take-all mechanism. (A) Architecture of the hippocampus network in de Almeida et al. (2009b, 2010). The granule cells in the dentate gryrus receives major convergent input from the entorhinal cortex. The pyramidal cells receive strong input from single granule cells and major convergent input from the entrohinal cortex. DG and CA3 exhibit E%MAX winner-take-all competition. (B) Competition is caused by the inhibitory interneurons that are capable of emitting global feedback inhibition (IPSP). (C) The interneurons are triggered after 3 ms of the first spike in the network cycle. Cells that are capable of reaching threshold during the 3 ms window also produce spikes, the other cells are inhibited before becoming active. Adapted from de Almeida et al. (2009a). nism solves the problem with the fixed linear threshold. Place cells from sensory input There are many models that try to explain the formation of place cells from sensory input. Many of these models are not constraint on biology but provide a robotic interpretation for the place cells phenomena. For instance, a computational model of context processing by Balkenius & Morén (2000) can produce spatial selectivity by considering location as a specific type of context. Rules of conditioning are used to associate the 3.1. COMPUTATIONAL MODELS OF SPATIAL SELECTIVITY 37 processed stimuli to position in a simulated robot. Further recall of the location of the robot is possible by comparing the current input vector with the contexts stored in memory. Other models try to associate sensory input with the perception of the motor activity. For instance, in the work of Burgess et al. (1997) a small robot with an ontop camera is used to associate local visual landmarks together with a path integration system. Learning is accomplished by hebbian rules. With this strategy they were able to emerge place cells like spatial selectivity. Other similar systems obtained the same results with other strategies for reinforcement learning (Arleo & Gerstner, 2000; Arleo et al., 2001). Although vision is the main source of sensory input, other modalities are also relevante. For instance, the model of Kulvicius et al. (2008) use odor cues to build a place cell network. One interesting approach is provided by the model of Verschure et al. (2006). They used a small robot with a ontop camera to explore a square area. It is different to the model of Burgess et al. (1997) because no motor information is used. The robot is steered by a simple random controller that alternates rotation with forward moves. They specified a neural network with 5 layers which are hierarchically organized. The layers are connected by convergent synapses. The layers are also provided with intra-area lateral connectivity. Upstream layers have more cells which are more sparsely connected. Downstream layers have less cells which are fully connected. The neurons are projected as mean-field units with local memory, i.e. activity is not only defined by the instantaneous input, it also includes a short trace from the recent input history. This network receives as its input a retinotopic projection of the camera input. The network is plastic and follows a learning rule that tries to optimize the 38 CHAPTER 3. COMPUTATIONAL MODELS OF THE MEDIAL TEMPORAL LOBE stability of single units (i.e. minimize the variability of the activity) and the diversity of activity in the same layer. This setup was able to emerge place cell like spatial selectivity in the highest layer after training. 3.1.3 Place cell navigation and behavior There exist many models of robot navigation based on hippocampal place cells. In general they are used to control small robots or simulated virtual agents. The key feature of most of these systems is the strategy used to link action to the activity of place cells. In the model of Gaussier et al. (2002), place cells are learned with supervision (to the neurons the precise location information is available and thus it just need to associate the sensory input). A cognitive map is build by linking the place cells using the delay between their activation during the learning phase. To allow action decision the system also defines a "transition cell" that is associated to the action that once took the robot from one position to the other. With this system in place, navigation can be obtained by chaining the place cells until the goal is reached and the actions can be decided on the basis of the needed transitions. The model of Foster et al. (2000) uses a temporal difference learning method in two distinct components: an actor-critic strategy and a network that uses temporal difference learning and self-motion information to acquire consistent spatial coordinates in the environment. The method was used to solve two navigation tasks: reference memory in the watermaze and a delayed matching-to-place task. Other models such as (Arleo et al., 2001) are able to learn action-value functions over a continuous location space, allowing the construction of 3.2. COMPUTATIONAL MODELS OF MEMORY 39 a navigation map.Using a similar strategy and the same Q-Learning algorithm, the model of Kulvicius et al. (2008) used odor-based place cells to guide navigation. 3.2 3.2.1 Computational models of memory Pattern completion and attractor dynamics The CA3 is a major inspiration for the formation of theoretical attractor dynamics based memory systems. The classic network from (Hopfield, 1982) was inspired on the massive lateral connectivity between the CA3 pyramidal cells and the observation of LTP in these synapses. The concept behind the attractor dynamics network is that the activity in the network will always converge to a stable state. There could be multiple stable states for a single system in a way that depending on the initial condition of the network the activity will converge to a different stable pattern, a.k.a. attractor. These stable states can be interpreted as memories (Marr, 1971). In the case that an incomplete input related to a memory is presented, the system will converge to the activity related to the perfect memory input. This process is known as pattern completion. There are however some requirements to a network system to work like the theoretical computational models. For instance, the time dynamics until the convergence is relevant. Also, it is predicted that the activity is persistent in the neurons. The real CA3 network does not have the same properties as the theoretical models. For instance, connectivity is far more sparse and there is no persistent firing given the intrinsic gamma modulation. However, some of the the theoretical features apply. A series of hippocampal models based on attractor dynamics is presented by Rolls (2007). 40 CHAPTER 3. COMPUTATIONAL MODELS OF THE MEDIAL TEMPORAL LOBE A work by de Almeida et al. (2007) has shown that, when considering the anatomical aspects of the CA3 network, it is possible to recall memories within a single gamma cycle. By means of computer simulation they could also estimate the size of the CA3 memory as 20.000 items. 3.2.2 Memory sequences in the hippocampus Major evidence for the existence of sequences in the hippocampus is the replay of previous behavior experience sequences during sleep (Louie & Wilson, 2001; Lee & Wilson, 2002) and awake states (Foster & Wilson, 2006) . A model by Lisman (1999) was able to associate sequences with the hippocampus structure. According to this model, sequences can be produced by the cyclic connectivity between the dentate gyrus and the CA3. Memories are formed in a single gamma cycle with the pattern completion process in the CA3. The activity of the CA3 neurons is projected back to the DG, which will cycle back to the CA3 as input to the next gamma cycle forming the next memory. The initial memory cue is obtained from the entorhinal cortex in the beginning of the theta cycle. This chaining of theta and gamma cycles allow storage of 7 ± 2 memories as observed on psychological studies Lisman & Idiart (1995). CHAPTER Mechanisms of conjunctive selectivity in the dentate gyrus Our starting point to better understand how place cells become conjunctive starts from the identification of the mechanisms that allow place selectivity. Much of the work presented in this chapter have major support on a series of papers published by Licurgo de Almeida, Marco Idart and John Lisman (de Almeida et al., 2009a,b) in which they identified many of the underlying mechanisms of place field formation in the dentate gyrus. Our major contribution was to use (and extend) these models to consider the nonspatial input of the Lateral Entorhinal Cortex. Supporting behavioral evidence comes from the fact that lesion of MEC impair spatial task and lesion of LEC impair spatial/nonspatial mixed task (Van Cauter et al., 2012). With this approach we could explain the phenomena of rate remapping (Leutgeb et al., 2007) and therefore provide a mechanistic description for the conjunctive response of place cells. This chapter reproduces the paper "The Mechanism of Rate Remapping 42 4 43 Rate map Population inhibition level Sum of MEC input Sum of LEC input position (x) position (y) Figure 4.1: Illustration of the process of place cell formation including LEC and MEC inputs. For each neuron, the excitement is computed for each position (x, y). Input from LEC is added to the input from MEC. At each specific position all cells compete through the E%-MAX process that outputs a population inhibition level. Rate map is build from the amount of excitation that exceeds the inhibition plane. This process is used in (Rennó-Costa et al., 2010a) and is analog to the one used in (de Almeida et al., 2009b, 2010). in the Dentate Gyrus", which was published at Neuron (Rennó-Costa et al., 2010a). The abstract reads: Rate remapping is a recently revealed neural code in which sensory information modulates the firing rate of hippocampal place cells. The mechanism underlying rate remapping is unknown. Its characteristic modulation, however, must arise from the interaction of the two major inputs to the hippocampus, the medial entorhinal cortex (MEC), in which grid cells CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 44 THE DENTATE GYRUS represent the spatial position of the rat, and the lateral entorhinal cortex (LEC), in which cells represent the sensory properties of the environment. We have used computational methods to elucidate the mechanism by which this interaction produces rate remapping. We show that the convergence of LEC and MEC inputs, in conjunction with a competitive network process mediated by feedback inhibition, can account quantitatively for this phenomenon. The same principle accounts for the independent variation of the place fields of the same cell as its sensory information is altered. Our results show that rate remapping is a novel code that can be explained in terms of known mechanisms. 4.1 Introduction Early work on the receptive field properties of rat hippocampal cells showed that their firing depends strongly on the rat’s location (O’Keefe & Dostrovsky, 1971). Indeed, their activity is generally restricted to one or several small regions of the environment called place fields. However, the hippocampus is also a storage site for non-spatial information (Wood et al., 1999; Rolls et al., 2005) so such information must somehow be represented. The fact that the spatial properties of hippocampal firing is modulated by manipulations of sensory cues (O’Keefe & Conway, 1978; Muller et al., 1991) and behavioral context (Wood et al., 2000) indicates that both spatial and non-spatial information are sharing the same neural structures and are likely to use a single common coding scheme. Recent work explored this question using a procedure in which the shape of the environment’s walls were slowly morphed from square to round (or vice versa), thereby changing their sensory qualities. It was found that such morphing changed the rate of firing of individual place cells, either upwards or downwards, a phenomenon called “rate remapping” (Leutgeb 4.1. INTRODUCTION 45 et al., 2005, 2007). Moreover, different place fields of the same cell can change upwards and downward independently. Thus, coding is not a cellular property, but the property of individual fields, each of which represents a separate conjunction of spatial and sensory information. This form of coding has not been previously observed in the brain and is very different from how sensory information is encoded in inferotemporal cortex, where cells represent specific sensory constructs, largely independent of their spatial position (Hung et al., 2005). Rate remapping, in contrast, permits the distinct representation of sensory events while maintaining the integrity of a code for spatial location. The mechanism underlying rate remapping has not been previously addressed. The hippocampus receives inputs from two regions of the entorhinal cortex (EC). One input is the medial entorhinal cortex (MEC), a region that contains grid cells of varying spatial frequency, orientation and phase (Hafting et al., 2005). The axons of many such cells converge on the dendrites of the granule cells of the dentate gyrus (DG), the first-order processing stage of the hippocampus. These granule cells show one or more place fields (Leutgeb et al., 2007). A previous computational study indicates that the summation of excitatory input from MEC grid cells, in conjunction with feedback inhibition from the dentate network, is sufficient to account for the spatially specific firing pattern of granule cells (de Almeida et al., 2009b). Moreover, this study showed that the realignment of the MEC grid cell population automatically makes the granule cells globally remap, as observed experimentally (Leutgeb et al., 2005, 2007). However, this mechanism alone cannot account for rate remapping because the MEC input itself does not change during environmental morphing (Leutgeb et al., 2007; Fyhn et al., 2007). Several lines of evidence indicate that sensory information about the environment is brought to the hippocampus by input from the lateral entorhinal cortex (LEC): in rodents, this region is itself driven by sensory related areas including CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 46 THE DENTATE GYRUS almost exclusive direct inputs from the ventral visual processing pathways of the occipitotemporal cortex (McDonald & Mascagni, 1996), the olfactory bulb (Carlsen et al., 1982) and indirect sensory input from area 35 of the perirhinal cortex (Burwell & Amaral, 1998; Burwell, 2000). Consistent with the sensory role of LEC, lesion of this region produces decreased investigation of novel objects (Myhrer, 1988). Furthermore, direct recordings from the LEC exhibit a spatial response with low selectivity, indicating the influence of the sensory (non-spatial) drive (Hargreaves et al., 2005). The inputs from the LEC converge with those from the MEC onto all granule cells of the DG. Since the LEC and MEC constitute the main source of the extra hippocampal input to the DG, it is this convergence that must somehow account for the rate remapping of DG cells. We have used computational methods to study the effects of these inputs from the EC onto the DG and have sought to answer two main questions: (1) What is the mechanism of rate remapping? (2) Why do different place fields of the same DG cell display independent rate remapping? 4.2 Results We simulated the response of DG cells to inputs from MEC and LEC in the following way. The spatial response (rate maps) of the grid cells were modeled as previously described (Blair et al., 2007; de Almeida et al., 2009b) and, in accord with data (Leutgeb et al., 2007), were made insensitive to morphing. 10 examples of such cells are shown in Figure 4.2a. LEC cells were modeled to be consistent with the finding (Hargreaves et al., 2005) that the firing rate of these cells carries little, but not zero, information about the position of the rat (Figure 4.2c, t = 0.9957, P = 2e−7 ). To account for the sensory consequences of morphing on LEC, we assumed that the spatial response of each cell is switched from one map to an independently generated one at some random point 4.2. RESULTS 47 during morphing (different assumptions are examined in Supplemental Text, Figure 4.3). The resulting receptive fields are shown for 10 LEC cells in Figure 4.2b. In order to approximate the response dynamics of the EC during environment morphing we generated the rate maps for both LEC and MEC (10,000 neurons each). To compute the excitatory input to each individual DG neuron we used a realistic number of inputs (1200 from the MEC and 1500 from LEC; see Methods) and summed them. Each synaptic input to the DG was taken from a population of randomly chosen entorhinal neurons, with the synaptic weight randomly assigned according to the synaptic weight distribution derived from the distribution of synapse sizes (de Almeida et al., 2009b) as determined by serial EM (Trommald & Hulleberg, 1997). The spatial distribution of firing of 10,000 DG granule cells was computed by applying, at each position, a winner-take-all interaction over the sum of excitation input. This winner-take-all process is governed by the so called, E%-max principle (de Almeida et al., 2009a) derived from the interaction of excitation with gamma frequency feedback inhibition, a form of inhibition known to exist in this brain (Bragin et al., 1995; Towers et al., 2002; Pöschel et al., 2002) and that synchronizes the firing of DG cells (reviewed by Bartos et al. (2007). According to this principle, the level of inhibition is set such that cells will fire provided their excitation is within 10% of the cell with maximum excitation. For these cells, their rate is proportional to where they fall in this 10% range. The value of 10% is computed from d/τm (de Almeida et al., 2009a), where d = delay of feedback inhibition and τm = membrane time constant, both of which have been experimentally determined. A previous study showed that the interaction of MEC input with this form of inhibition is able to quantitatively account for the size and number of place fields exhibited by active DG cells (de Almeida et al., 2009b). In our simulations, we also take into consideration the LEC. The interaction of the two inputs depends on the ratio (α) of the mean drive of MEC and LEC onto EC. No data is available that would allow us to directly estimate α. However, our results provide for a quan- CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 48 THE DENTATE GYRUS titative estimate of its value (see below). With this simulation framework in place, we investigated whether the cumulative decorrelation of population output from the dentate gyrus observed during progressive morphing of the arena shape (Leutgeb et al., 2007) (PV correlation curve, Figure 3a) could be explained by the changes of the LEC spatial response. We computed the correlation between composite population vectors (see Methods) as a function of morphing stage throughout over a range of α and compared this with the correlations reported by Leutgeb et al. (2007) (Fig1d). To account for the variability of the firing rate in consecutive recordings under the same conditions (Hargreaves et al., 2005; Leutgeb et al., 2007; Fyhn et al., 2007), we emulated the effect of under-sampling of the space, an unavoidable condition given the experimental protocols. To account for the effect of undersampling, we introduced a stochastic factor in every comparison with a variance dependent on the rate (see Methods). The level of the correction was obtained by fitting to the experimental data (PV correlation) of two subsequent recordings obtained under the same condition (Figure 4.8). We observed an exponential-like decay shape for the correlation curves with the global level of decorrelation monotonically and positively affected by the level of influence of the LEC input (regulated by α). A value of (α=0.32, Figure 4.2) gave the best fit. With the value of α determined, we could then examine how morphing affected rate remapping. First, we investigated whether the simulated place fields have properties that match those experimentally observed. We found that simulated granule cells have multiple place fields (average of 2.2 place fields) and have a mean place field size of 943cm2 . The distribution of the number of place fields in each active cell was similar to experimental measurements (Figure 4.2e, t = 0.98, P < 0.0005). The place field size is also in accord with data (analysis of (Leutgeb et al., 2007) by de Almeida 4.2. RESULTS 49 Figure 4.2: MEC and LEC inputs and estimation of model parameters. (a) Example of the 10 MEC modeled rate maps (number is the maximum firing rate). MEC rate maps remain constant during morphing. (b) Example of the 10 LEC rate maps from experimental data (H, maximum rate when informed, adapted from (Hargreaves et al., 2005) and 10 from the model for the two environments (square and round, maximum rate in both environments). Rate maps presented with equally distributed spatial score (ranked from right to left). (c) Histogram of spatial information score from LEC rate maps (H, experimental data and square, model. correlation 0.9957, P < 0.05). (d) Ratio (α) between the mean firing rates in MEC and LEC estimated as 0.32 by fitting to the experimentally observed reduction on spatial coincidence using population vector correlation as the environment is morphed (Leutgeb et al., 2007). (e) Histogram of the number of place fields found in DG cells (Leutgeb and square environment). Stable high correlation between experimental and simulated histograms during morphing indicates that modification in LEC activity do not disrupt place field formation (R = 0.98). CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 50 THE DENTATE GYRUS α PV Correlation 0.9 0.9 MEC 0.7 0.7 0.5 0.5 0.3 0% Interpolated Abrupt (a = 0.32) [Leutgeb 2007] 0.3 0.2 0.0 LEC 17% 33% 50% 67% 83% 100% Morphing progression Figure 4.3: Difference in spatial coincidence reduction rate for abrupt and linear interpolated morphing of LEC spatial response. Comparison of the mean population vector (PV) during remapping compared with (Leutgeb et al., 2007). For 17% of morphing, the minimal correlation value is 0.82 ± 0.01 compared to 0.75 observed experimentally. et al. (2009b)). We also tested whether the observed restricted diversity of grid cell activity (Barry et al., 2007) affects the results of our simulation. When the grid cell proprieties were limited to one orientation and three grades of spacing, no significant difference in the distribution of the number of place fields (Wilcoxon, p = 0.65) or the PV correlation (Student t-test two-tailed, p = 0.31) was found. These results are not unexpected given previous work showing that MEC input alone can account for these properties; what is added here is the demonstration that the LEC inputs, when included in the model, do not interfere with place cell formation in the DG by the MEC inputs. We next directly compared the remapping of individual place fields of our simulation of morphing with the results obtained by Leutgeb et al. (2007) 4.2. RESULTS 51 (Figure 4.4a). The experimental results show that all place fields of the same cell remap and do so independently; thus one field may increase its firing rate during morphing while the other decreases its rate. Figure 4.4b shows this to be similarly true in our simulated place fields. Moreover, the relative proportion of remapping patterns (linear, quadratic and sigmoidal) could not be distinguished from the experimental observations (Figure 4.4c, t = 0.93, not significant (n.s.)). To obtain insight into why remapping is independent for different place fields of the same cell, we analyzed the changes during morphing (Figure 4.5). We identified two processes that cause independent place field rate remapping: (A) the effect of morphing on LEC cells changes the direct excitation of the granule cells (Figure 4.5A). Since the rate change of LEC cells due to morphing is a function of position, the variation on the integration of the LEC excitatory input is independent for each place field; (B) the change of the excitation of other cells will determine which cell is most activate at a given position. This determines the E% max level and thereby indirectly, via inhibition, alters the rate of other cells (Figure 4.5B). This process is localized and therefore independent for each place field. To determine which mechanism prevails in controlling rate remapping, i.e. excitatory drive versus inhibitory competition, we looked for the ratio between the levels of remapping accounted for by each mechanism (see Methods). We observed that both mechanisms contribute to almost all place fields, with a slight dominance of mechanism A (Figure 4.6). CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 52 THE DENTATE GYRUS Figure 4.4: Simulated DG cells exhibit independent place field rate remapping, as observed experimentally. Differential rate changes in individual firing fields of cells from the dentate gyrus during progressive maneuvering of the walls of the arena. (a) Recorded cells. Adapted from (Leutgeb et al., 2007). (b) Simulated cells. Individual fields are numerically labeled to relate to the respective line diagram of the mean field rate. The rate curves were fitted to linear (red), quadratic (green) or sigmoid (blue) functions and are shown when significant (p < 0.05, dotted line). (c) Histogram of the best fit classification for recorded and simulated curves. Correlation between histograms is of 0.9543 (P = 0.045). 4.3 Discussion Rate remapping is a new form of coding, the mechanism of which has been unclear. We have found that it can be explained in terms of simple processes: the summation of several thousand LEC and MEC inputs to DG cells, in conjunction with a network process that produces com- 4.3. DISCUSSION 53 Figure 4.5: Different mechanisms for independent rate remapping of different place fields of the same cell. (A) Rate is directly affected by changes of the input drive. For a given cell, morphing (round to square) induces localized variation of LEC input, changing the rate of each place field independently (At PF1 , elevation of input drive (INPUT1 ) causes the rise of rate (RATE1 ). At PF2 , the fall of the input level (INPUT2 ) leads to reduction of rate (RATE2 )). In this case, remapping is only caused by the change of the input since the global inhibition level does not vary (dotted red line); (B) Rate is inversely affected by changes of the inhibition. Morphing induces localized variation of the global inhibition level, changing the rate of each place field independently (At PF1 , the raise of the global inhibition level (INH3 ) causes the decay of the rate (RATE3 ). At PF2 , the fall of the global inhibition level (INH4 ) causes the rise of the rate (RATE4 )). In this case, remapping is only caused by the local changes on the global inhibition level since all inputs to this cell remain in the same level during remapping. The change of the inhibition level is caused by variations of the input drive of the most excited cell. For each cell and wall shape a rate map is shown with the relevant place fields indicated by a white circle and the process values used of the computation of the rate at these place fields: the sum of entorhinal input (light gray bar for LEC and dark gray bar for MEC); the sum of entorhinal input of the most excited cell (red line); the global inhibition level (dotted red line) and the rate (black bar). CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 54 THE DENTATE GYRUS Figure 4.6: Distribution of the mechanism balance ratio through active place fields. For clarification see Methods. Low ratio indicates prevalence of first mechanism (Figure 4.5A) while high ratio indicates that second mechanism (Figure 4.5B) is more effective. petitive inhibition. These mechanisms are sufficient to explain the key observation, that even though the LEC input to the DG is not restricted to specific positions, virtually all DG cells have place fields. Our simulations show that the spatial firing pattern of DG cells is determined primarily by the MEC inputs; the role of the LEC is to determine the specific rate at which place cells fire. In addition to accounting for these findings, our model elucidates several other properties, notably the size of place fields, the average number of place fields, and the fact that if DG cells have multiple place fields, these vary independently during morphing of the environment. Other models have investigated the integration of input from LEC and MEC in the DG (Hayman & Jeffery, 2008; Si & Treves, 2009) and provided some insights that are consistent with our results. However our model is the first to attempt to quantitatively account for rate remapping (for a comparison of models, see Supplementary Text). The mechanism of rate remapping can be understood intuitively in terms of the summation of LEC and MEC inputs and the strong competition for firing in DG produced by the DG inhibitory network (Figure 4.5). 4.3. DISCUSSION 55 In this context, the strength of an input is defined by the presynaptic activity of the neurons of the entorhinal cortex and the strength of the synapses they form onto granule cells. If only the most excited cells can fire, then cells with both strong LEC and MEC input will have great advantage in this competition. Thus, only cells that have strong MEC inputs, and are thus “successful” place cells, can express the additional input from the LEC. Conversely, cells that have strong LEC input, but weak MEC input, and which could therefore express properties of the sensory world largely independent of place, are unlikely to be winners. This explains why cells that solely code sensory information, like those in the LEC and IT cortex, are very rare in the DG. This implies that the representation of the environment, as conveyed by LEC, is mixed in the DG with the spatial metric imposed by MEC. Although convergence and competition are keys to understanding the mechanism of rate remapping, two additional factors should be noted. First, the number of inputs into a single DG cell from both LEC and MEC are large (>1000) and therefore not subject to large statistical fluctuations. If the number were much smaller, it might often arise by chance that significant numbers of DG cells received strong enough LEC input to win the competition even with negligible MEC input, contrary to what is observed (see Supplemental Text, Figure 4.7). Second, spatial encoding is unique because the organism is always at a place; i.e. the MEC is always active and formation of grid cells is not impaired by darkness (Hafting et al., 2005). In contrast, information from any specific sensory modality in the LEC may be present or not at any point in time. Because place is always present, other sensory information can never compete by itself for influence over the DG; the competition is always influenced by MEC input. It may happen that sensory input affects the properties of the grid cells when grids realign to distal cues (Sargolini et al., 2006), but such changes only occur during global remapping, which is outside CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 56 THE DENTATE GYRUS the scope of this study. The mechanism we propose for rate remapping depends on the interaction of the LEC and MEC. This interaction depends quantitatively on the relative magnitude of the two inputs (α), which according to our analysis should be in the range of 0.2 − 0.3. Importantly, modification of α provides a way of testing the proposed model of rate remapping. Specifically, (1) the mean population vector correlation produced by morphing should monotonically increase with α (Figure 4.2d) and (2) the mean place field size should monotonically decrease with α (Figure 4.7d). With the advent of molecular methods for altering firing rates or synaptic strengths in a region specific manner, it should become possible to directly test these predictions. Previous studies have shown that multiple place fields of single DG neurons emerge from the mechanism considered here using inputs from MEC only (de Almeida et al., 2009b). Our simulations show that this phenomenon still holds when inputs from both MEC and LEC are considered. What emerges from our analysis is that simple random summation of the inputs and competition among DG cells is sufficient to form place fields, but not selective enough to form only one; i.e. multiple place fields is the best the system can do in decoding the highly distributed grid cell input. The emergence of cells with single place fields, as occurs in CA3, requires an additional processing step (de Almeida et al., 2010). The independence of the rate remapping observed in the multiple place fields of single DG cells (Leutgeb et al., 2007) constitutes a novel form of neural code. In this code the DG neuron multiplexes multiple independent features that are selected on the basis of a spatial metric. The independence emerges because both excitation and inhibition vary with spatial location. Rate remapping is different from other rate codes in 4.3. DISCUSSION 57 the brain that are selective for multiple features, as for instance, the combined spatial frequency and orientation tuning curves found in single neurons of the primary visual cortex (De Valois & De Valois, 1990). The overall response of these V1 cells can be explained by the multiplication of tuning curves that, in contrast to the rate remapping in the DG, are fixed and invariant to any other feature change (Mazer et al., 2002). The independent (nonmultiplicative) modulation of the place fields of single DG neurons promotes orthogonalization of the encoding that is required to generate the highly specific responses to single locations found in CA3 (Leutgeb et al., 2007). Our results answer some questions about this code, but other important questions remain. A defining feature of this code is that the firing rate is not binary. Thus, a particular memory is represented not only by which cells fire, but also by the firing rates. Now consider the process of pattern completion for n cells with rates R1, R2. . . Rn. Suppose a partial cue is presented, say R1 to R5. This should lead to the firing of unstimulated cells at their appropriate graded rates. Indeed, there are attractor network models that use graded rather than binary rates (Rolls, 2007), and it will be interesting to see if these can account quantitatively for pattern completion in CA3. Another unanswered question is where and how rate remapping is decoded so that cortical cells, which do not code sensory information using spatially specific cells, can decode information (such as during replay) that they receive from the hippocampus. CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 58 THE DENTATE GYRUS 4.4 4.4.1 Supplemental material Alternative assumptions about how the LEC responds to morphing In the main text we assumed that the spatial response of LEC cells in two arenas with different shapes is uncorrelated. We also assumed that during morphing, the spatial response is abruptly changed at some random point during morphing. To test whether other assumptions would lead to similar results, we analyzed two alternatives: (1) the spatial response of LEC cells is highly correlated in the two arena shapes, having only the mean fire rate varying with the sensory condition; (2) the spatial response of LEC varies linearly with morphing by means of interpolation. In the first alternative, a base rate map was created invariant to morphing for each cell (see Methods). To the base spatial response was added a positive bias that linearly relates to the morphing degree, having its value at the extremities of morphing defined randomly. This preserves the spatial correlation of the spatial map but changes the mean drive of each cell. With this form of LEC encoding, the minimum PV correlation value obtained (0.71 ± 0.01) was significantly higher than the minimum experimental value (0.32), rejecting this class of information coding in LEC. In the second alternative, two base rate maps were created for each cell exactly as in the main text. However, instead of changing abruptly which rate map is effective at a random point of morphing, the effective rate map was a linear interpolation of the two base rate maps following the morphing stage. The change in rate was proportional to the degree of morphing, making this a form of rate remapping in the LEC. Although the PV correlation values were within the experimental range of the two extremes of the morphing, the inclination of the curve was significantly 4.4. SUPPLEMENTAL MATERIAL 59 different; it showed a smaller decay of the PV correlation of the first stage of the morphing (Figure 4.3). 4.4.2 Differences in how DG and LEC encode sensory information To test whether DG cells have a similar representation than LEC cells we examined how the LEC and DG cells code sensory and spatial information. Using as a metric for the amount of information that each spike carries about the position and the shape of the arena (Skaggs et al., 1993) we observed two clusters (Figure 4.7a). DG cells presented significantly higher spatial information score (Figure 4.7b, DG, median = 2.02 bits/spike, LEC, median = 0.16 bit/spike, Mann-Whitney P < 0.05 twotailed) and higher shape information score (Figure 4.7c, DG, median = 0.14 bit/spike, LEC, median = 0.09 bit/spike, Mann-Whitney P < 0.05 two-tailed). This illustrates that LEC cells do not impose their character on hippocampal cells (there are no “round” or “square” cells that fire over a large fraction of the environment). Rather the firing of hippocampal cells represents sensory information in a very spatially restricted fashion. 4.4.3 Comparison to other models The integration of LEC and MEC and both types of remapping that occur in the hippocampus have been the target of previous modeling efforts. Si & Treves (2009) presented a model in which place fields are formed by applying a competitive Hebbian learning strategy to a postsynaptic target driven by a population of simulated grid cells. They successively produce place fields with this strategy and, by including the LEC input as an overall rate bias, both papers reached similar conclusions conclusions on how the proprieties of place fields are affected: larger place fields (Figure 4.7d) and a higher number of place fields (data not shown). This CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 60 THE DENTATE GYRUS model, unlike the one presented here, however, does not consider any specific anatomical constraints and requires learning. Moreover, the authors do not mention how remapping affects their model, but they probably can simulate some aspects of rate remapping by changing the values of their LEC input in a mechanism that is close to our mechanism A. Their model cannot reproduce the independent remapping of different place fields of the same cell because their LEC input is uniform with respect to position and thus cannot account for the data of Hargreaves et al. (2005). The model of Savelli & Knierim (2010) is also based on competitive Hebbian learning. It, like our previous work (de Almeida et al., 2009b) shows how global remapping could be produced by changing the alignment of the MEC grid cells, as observed by Fyhn et al. (2007). The drawback of the model of Savelli & Knierim (2010) is that, whenever the MEC grid cells realign, the plasticity mechanism changes the synaptic weights. This causes that when the rat is placed back to a previously known environment, “even if the grid cells fire in the same locations as in the first exploration”, “the model would produce a different set of place fields”, which contradicts the experimental observations that the original set of place fields is recovered once the rat is placed back in the familiar environment (Leutgeb et al., 2005; Fyhn et al., 2007). Since no plasticity is involved in our model, one specific grid cell alignment will always generate the same set of place cells. Hayman & Jeffery (2008) suggested that LEC and MEC inputs could be clustered on separate branches of the dendritic trees of the granule cells of the DG. In this model, each branch would integrate the activity of some subset of aligned grid cells, generating a place cell response. Subsequently the input from LEC would gate which branches are active. This model succeeds qualitatively in explaining some aspect of (partial) rate remapping. The gradual change of the LEC input could perform an 4.5. EXPERIMENTAL PROCEDURES 61 interpolation of several rate maps associated with a single cell, creating the rate remapping effect. However, such models have some limitations: (1) the diversity of contexts that evoke global remapping represented by each cell is limited by the number of clusters it can have; (2) the number of clusters is limited by the number of branches in the dendritic tree; (3) in the advent of a new environment, it might be required that one cell forgets previous clusters or that new cells are recruited. The first case will lead to the same dilemma as for the model discussed in the previous paragraph, while the second case has no experimental support (Leutgeb et al., 2007). In contrast, in our model there is practically no limitation to the diversity of contexts and the recruitment of new cells is in accordance to the literature (de Almeida et al., 2009b). 4.5 Experimental procedures Spatial response representation All data was simulated for a 1 m square enclosure with a resolution of 1 cm2 , compromising 10000 square bins organized in a 100 x 100 rectangular grid. The spatial response for each cell of all considered cortical regions was composed by rate values assigned for each bin, defining a rate map. Simulation of spatial response from entorhinal cortex MEC spatial response was set invariant to the morphing of the environment, being simulated only once. The rate (λ) of each MEC cell follows the equation defined by Blair et al. (2007) and is a function of the Cartesian position (r = (x, y)) and subject to the following parameters: the place field decay constant (a, normally distributed with 0.55 ± 0.03), the inter-vertex distance (d, ranging from 30 to 100 cm), the spatial offset (c CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 62 THE DENTATE GYRUS = (x0 , y0 ), ranging from (0, 0) to (d, d)) and the angular offset (θ, from 0o to 60o ): 4π √ (cos(θk +θ),sin(θk +θ))·(r−c) k=1 3d ! a· λ(r, a, d, c, θ) = e Pa − 3 2 ! − 1 (4.1) The vertex angles (θ1 = −30o , θ2 = +30o and θ3 = +90o ) define a honeycomb grid that bases the formation of the grid cell firing. We simulated the spatial response of 10000 MEC cells, each of them with a random parametric set within the range specified above. LEC spatial response was set dependent to the degree of morphing (v). Indeed, morphing was incorporated in the model by changing the spatial response of LEC cells. For each LEC cell there were assigned one rate map for the beginning and another for the end of the morphing, each of them generated independently (following the methods below). For the intermediate morphing steps, it was defined a random (uniformly distributed) transition morphing degree for each cell in a way that the spatial response of the cell is invariant from the beginning to this point and from this point to the end. To synthesize the LEC rate maps, the arena was divided in a 5x5 grid that covers the whole arena. For each rate map, these regions were randomly separated in two groups (active or inactive) according to the expected spatial information score (high spatial specificity renders less active regions). A base rate map is built by assigning a random rate value within the range [0, 0.5] for non-active regions and [0.5, 1] for active regions. To obtain the final map of LEC responses we convolved the base map with a Gaussian kernel with standard deviation of 17 bins. We simulated 4.5. EXPERIMENTAL PROCEDURES 63 the spatial response of 10,000 LEC cells by using the number of active regions to fit to the experimental spatial information score (Hargreaves et al., 2005). Samples of LEC rate maps and the spatial information score histogram are shown in Figure 4.2b and Figure 4.2c respectively. LEC and MEC spatial responses had the population mean average rate normalized. Since we could not obtain information about the relative mean fire rate of MEC and LEC populations, we had the ratio parameterized by α in the range [0, 1] when the rates were integrated in the computation of the excitatory input of the granule cell. Granule cells Each granule cell integrates the excitatory input received from a random group of MEC and LEC cells following the estimated convergence (see below). The sum of entorhinal input of each granule cell (I) is specific for each position, which allow a map representation. The excitatory input is the product of the rate (λ) of the afferent cell with the specific synaptic weight (W , see below): Iiv (r) = α M EC X j λj (r) · Wij + (1 − α) LEC X λvk (r) · Wik (4.2) k The rate of granule cells is defined by competition of the sum of the entorhinal input within the population ruled by a percentage of maximal suprathreshold excitation (E%-max) winner-take-all process (de Almeida et al., 2009a), measured as 10%. At a specific position and arena shape, the amount of inhibition is equal to 90% of the sum of the entorhinal input of the most excited cell in the population. Whenever this global CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 64 THE DENTATE GYRUS inhibition is higher than the sum of entorhinal inputs of a specific cell, this cell remains silent. Otherwise, the rate of the cell is the difference between excitation and inhibition. v v DG v λvi (r) = Iiv (r) − 0.9 · maxDG I (r) ·H I (r) − 0.9 · max I (r) j j i j j (4.3) , where H is a Heaviside function. Figure 4.3 gives an insight of how granule cells rate map is obtained from grid cells and LEC cells and how rate is influenced by both the input of entorhinal input of the cell and by the population inhibition. Convergence from entorhinal cortex The convergence of the entorhinal cortex input onto granule cells was estimated by the number of synapses as 1200 for grid cells (de Almeida et al., 2009b) and following the same procedure as 1500 for LEC inputs (see Supplementary Methods). Synaptic weight Synaptic weight (W ) is defined by the synaptic size (s) (de Almeida et al., 2009b): W (s) = s s 0.2 s + 0.0314 (4.4) The synaptic size distribution was defined by the measured size distribution of excitatory synapses onto granules cells (Trommald & Hulleberg, 1997): 4.5. EXPERIMENTAL PROCEDURES − P (s) = 100.7 1 − e 65 s s s − − 0.022 · e 0.018 + 0.02 · e 0.15 (4.5) , s ranges from 0 to 0.2. Data analysis Cells with average firing rate above 10% of the mean average firing rate of cell population were considered active. Composite population vector correlation Composite population vector (PV) correlation has been used in the analysis of experimental data to observe the reduction of rate coincidence at the same position in the dentate gyrus when the shape of the arena is morphed (Leutgeb et al., 2007). PVs are obtained by storing in a vector the rate at a certain position bin of each cell of a population. The correlation between the PV of the same group on two different conditions give a measure of how the condition affects the overall population activity. The PV correlation value is the mean correlation value considering all positions bins. Place field analysis The number of place fields was estimated from the rate map for active cells in each stage of the morphing. Rate maps were smoothed by a Gaussian kernel with 9 pixels radius. Pixels with firing rate above 20% of the peak rate were considered active. Groups of contiguous active pixels (> 200 and < 2500pixels) with average rate exceeding the mean population fire rate and with peak activity above two times the mean CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 66 THE DENTATE GYRUS population fire rate were considered to be a firing field. Curve fit Persistent place fields were obtained by applying place field analysis on the average rate map for all morphing shapes (Leutgeb et al., 2007). There different curves were fit to the in-field rate for each persistent place field following the morphing: (a) linear regression, (b) quadratic regression and (c) sigmoid function. Fits with p values < 0.05 were considered significant, and each place field was assigned to the category with the highest explained variance (F values). Rate remapping measures The level of the rate remapping effect is measured for each persistent place field (p) whose average mean rate for the two extreme shapes of the morphing (λSR ) is above 10% of the mean average firing rate of the cell population. The rate remapping level (ηR ) is defined as the absolute difference in firing rate normalized by λSR . The level of rate remapping due to mechanism A (ηA ), which is based on the change of the sum of direct excitatory inputs, is the absolute difference in the mean sum of the input at the positions of the place field normalized by λSR . The level of rate remapping due to mechanism B (ηB ), which is based on the change in the level of inhibition, is the absolute difference in the mean global inhibition level at the positions of the place field, normalized by λSR . The ratio of the impact of the two mechanisms (γ) is ηB divided by ηA + ηB . Pr⊂p ηR (p) = r |λ1i (r) − λ0i (r)| Pr⊂p λSR r (4.6) 4.5. EXPERIMENTAL PROCEDURES 67 Pr⊂p ηA (p) = Pr⊂p ηB (p) = r r |Ii1 (r) − Ii0 (r)| Pr⊂p λSR r 1 DG 0 |0.9 · maxDG j (Ij (r)) − 0.9 · maxj (Ij (r))| Pr⊂p SR λ r γ(p) = ηB ηA + ηB (4.7) (4.8) (4.9) Estimation of the convergence from entorhinal cortex For the projection from the grid cells the number of synapses has been previously estimated as 1200 (de Almeida et al., 2009b). The number of synapses made by LEC cells onto granule cells can be estimated following the same methods: granule cells have 3000 µm (Johnston & Amaral, 1998) of dendrite and spine density of 2.3 spines/µm (Johnston et al. 1998); Each spine has one synapse; There are thus 6840 synapses; The middle molecular layer has 3̃0% of the dendrite area while the distal layer has 25% (Hama et al. 1989); 85% of the synapses receive input from the layer 2 of the entorhinal cortex (Nafstad, 1967); Since the fraction of silent synapses is small in this cell type (Min et al., 1998) there are 1500 LEC inputs. Emulation of experimental condition Simulated rate maps take into consideration mean fire rates that are obtained by a deterministic mechanism and therefore are always the same. Under experimental conditions, however, the observed rate is subject to a variance caused by non-uniform sampling of the space and the non deterministic nature of spiking neurons. This is the most straightforward explanation for why two successive rate maps obtained under the CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 68 THE DENTATE GYRUS same environmental conditions are not always the same, as observed for LEC and MEC rate maps (Hargreaves et al., 2005) and for the rate of grid fields (Fyhn et al., 2007). In Leutgeb et al. (2007), two successive recordings of the same neural population showed a PV correlation vector of 0.87. To emulate this experimental factor in the calculation of the PV correlation of simulated data, we used random rates from a normal distribution with the model rate value as mean and a specific variance. The variance (var(r)) of a certain bin was set to be proportional to the mean rate (< λ(r) >) at this position, since the non-deterministic event observed is spikes and therefore the number of probabilistic events is proportional to the rate, and to an experimental factor (β): var(r) = β · hλ(r)i (4.10) To determine β for the experimental conditions of Leutgeb et al. (2007), we fit the PV correlation value for two rate map groups for the same morphing stage (Figure 4.8). We observed that β varied with the number of cells used, with the PV correlation and with α. Spatial and shape information score The spatial information score is a reliable measure of the sharpness of spatial tuning of a spike train (Skaggs et al., 1996). The spatial information score has been derived from communication theory by considering that a cell is a communication channel with the rat’s location as input and cell’s spike activity as output, assuming that all information about position is encoded by the fire rate (Skaggs et al., 1993). The spatial information score of rate map depends on the occupancy probability (p(r), sampling time at position r relative to the total sampling time), the rate at each position (λ(r)) and the mean rate (< λ >): 4.5. EXPERIMENTAL PROCEDURES SpatialInf ormation = bins X 69 p(r) r λ(r) λ(r) log2 hλi hλi (4.11) Shape information score follows the same method above by considering as input the shape of the environment and the output the cell’s spike train, assuming that all information about the shape of the arena is encoded by the fire rate. It is a measure of the sensory tuning of the spike train independent of the position of the rat. CHAPTER 4. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 70 THE DENTATE GYRUS A B 5 4.5 DG LEC Spatial info (bits/spike) Shape info (bits/spike) 1 0.8 0.6 0.4 4 3.5 3 2.5 2 1.5 1 0.2 0.5 0 0 1 2 3 4 C 0 5 Mean spatial info (bits/spike) D LEC DG 1200 0.9 1000 0.8 2 Mean place field size (cm ) Shape info score (bit/spike) 1 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 LEC DG 800 600 400 200 0 0.0 0.1 LEC 0.2 0.3 0.4 0.5 α 0.6 0.7 0.8 0.9 1.0 MEC Figure 4.7: DG cells do not represent sensory and spatial information in the same way as LEC cells. (A) Plot of the mean spatial information score on both shapes and the shape information score of LEC cells (blue, x) and DG cells (red, o). Spatial information score measures quantitatively how the position is encoded by one spike while shape information score relates to how much information about the shape each spike carries. (B) Box plot of the mean spatial information score and of (C) the shape information score for both populations. In each box, the central mark is the median, the edges of the box are the 25th and 75th percentiles and the whiskers extend to the most extreme data points. (D) The relative contribution of LEC and MEC input influences spatial properties. Histogram of the mean place field size as function of the ratio (α) of the mean drive of MEC and LEC onto EC. Low alpha indicates high LEC influence while low alpha indicates stronger MEC input. 4.5. EXPERIMENTAL PROCEDURES A 71 B 1 66 Cells 3 [Leutgeb et al., 2007] Experimental factor (β) PV correlation 0.9 0.8 0.7 10000 Cells 3 2 2 1 1 66 Cells 0.6 10000 Cells 0.5 1 1.5 2 Variance/Frequency 2.5 3 0 0 LEC 0.5 α 1 MEC 0 0 LEC 0.5 α 1 MEC Figure 4.8: Experimental variance correction for simulated data. (A) Mean population vector (PV) correlation for two successive recordings of the same morphing stage decays linearly with the increase of frequency proportional variance. The number of cells considered for the PV influences the effect of variance in correlation: less cells raises sensibility. To correct the simulated data to the experimental condition of (Leutgeb et al., 2007) we used the variance/frequency value (experimental factor β) that fits the experimental PV correlation. (B) Fitting of β is influenced by both number of cells and LEC/MEC mean rate ratio. CHAPTER Mechanisms of conjunctive selectivity in the CA3 The content of this section has not yet been published and is part of a manuscript in preparation with the cooperation of John Lisman and Paul Verschure. The manuscript is named: "The mechanism of attractor dynamics in the CA3". The aim of this study is to explain the mechanism of rate remapping in the CA3 and expose its implications in the understanding of how attractor dynamics work in the CA3 network. Our approach was to extend the model of de Almeida et al. (2010) by incorporating the LEC input and by adding the time-constraints of the recurrent synapses in the CA3 (Figure 4.1). The abstract has been published as a Society for Neuroscience abstract (Rennó-Costa et al., 2012b): Attractor networks are thought to play a fundamental role in memory by producing pattern completion. A characteristic feature of such networks is their recurrent connectivity. Given its specific connectivity the hippocampal CA3 region has been suspected to follow attractor dynamics. However the exper72 5 5.1. INTRODUCTION 73 imental evidence supporting this notion has not been conclusive. Here we analyze the phenomenon of rate remapping that occurs as the environment is gradually morphed (Leutgeb et al., 2007). During such morphing, the grid cells that provide spatial localization input to the hippocampus are unchanged, explaining the constancy of place field location. Other inputs, however, provide sensory input affected by the morphing and combine with the spatial information to produce rate remapping in the firing rate of the place fields. A key finding is that CA3 place cells are less variant to small changes in the sensory input than those found in the Dentate Gyrus (DG). This is seen as an attractor-like effect. We have used computational modeling (Rennó-Costa et al., 2010a; de Almeida et al., 2010) to elucidate the origin of this difference in the physiology of DG and CA3. We find that the difference can be accounted for quantitatively by the fact that CA3 has recurrent connections, whereas the dentate gyrus does not. Furthermore, we show that the observed hysteresis in CA3 place cells suggests that changes in the attractor dynamics can be produced by continuous plasticity during the experiments. The ability to analyze the responses to morphing in the hippocampus thus provides a rather direct view of how attractors function in a biological memory network. 5.1 Introduction The major difference between rate remapping in the DG and in CA3 is illustrated by the PV correlation curve (Leutgeb et al., 2007, Figure 2A or in this text, Figure 2.7E). In the DG, the ensemble population change in PV correlation in relation to the recording in the first arena (1) is significantly higher for the first morphing stage (2) than the change to a second recording in the first arena after the whole morphing protocol CHAPTER 5. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 74 THE CA3 (1’). In contrast, in the CA3 there is no significant difference between the the 1-1’ and 1-2 PV correlations. This can be interpreted as an evidence of pattern completion, a typical feature of attractor dynamics and associative memory systems that is commonly attributed as an hippocampus function (Rolls, 2007; de Almeida et al., 2007). Pattern completion is the ability of a network to exhibit a stable output in the case of an incomplete input. It is a feature of attractor networks that are able to converge the output to a stable attractor if the input is within the attractor neighborhood Amit (1992). The matching of 1-1’ and 1-2 correlations might suggest that the CA3 interprets that the first two shapes (1 and 2) are indeed the same shape. Thus, understanding the differences of dynamics in the rate remapping phenomenon might ultimately explain how pattern completion is accomplished in the hippocampus. The mechanism of pattern completion is likely to be within the CA3 network given that its major input, the DG, is able to distinguish the two arenas. The major difference between DG and CA3 principal cells is that CA3 cells present recurrent connectivity, a major feature of the neural networks models of attractor dynamics Hopfield (1982). Recurrent connectivity can pattern complete by setting ensembles of cells. Each cell of the ensemble do not only receive input from upstream regions but also from other cells of the same the ensemble. In the case that a noisy version of a memory is presented to the network, it is likely that some cells of the ensemble will not receive the expected input. However, the excitation originated in the other cells of the ensemble will be sufficient to activate these cells, allowing the whole ensemble to be active (de Almeida et al., 2007). Ensembles could be formed to represent a specific input pattern by auto-associative learning rules (Hebb, 1932). Although the ensembles might allow pattern completion, the nature of the time dynamics of the CA3 networks creates very restrictive constrains 5.2. RESULTS 75 when compared tom theoretical models. For instance, the recurrent excitation must occur within the feedback inhibition delay of 3 ms. Moreover, given that inhibition is evoked by the network, it is likely that competition modulates the pattern completion process. This allows very low bandwidth for the ensemble process forcing the memory recall to happen in a single interaction and not by sustained activity (de Almeida et al., 2007). To better understand the dynamics of the recurrent connections we implemented a time sensitive spike-based neural network implementing the same principles of previous rate model: massive input convergence and feedback inhibition. We base our implementation in standard integrate and fire neurons with parameters extracted from CA3 physiology (de Almeida et al., 2007) and in the network connectivity of the CA3 (de Almeida et al., 2010). With the same experimental protocol as previous session, we observed how the change in LEC population affected the overall population activity in the CA3. 5.2 Results In a first step we simulated the response of DG cells to the input from LEC and MEC by adapting methods from previous section (Figure 5.1A). The spatial response (rate maps) of 10,000 grid cells were made insensitive to the shape of the arena (Leutgeb et al., 2007), whilst 10,000 LEC cells had their spatial response switched from one map to an independent one at some random morphing progression, specific for each cell. To compute the excitatory input to each individual DG cell we used a realistic number of inputs and summed them. Each synaptic input was taken from a population of randomly chosen entorhinal neurons, with the synaptic weight randomly assigned according to the synaptic weight distribution derived from the distribution of synaptic sizes (de Almeida CHAPTER 5. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 76 THE CA3 et al., 2009a) as determined by serial EM (Trommald & Hulleberg, 1997). Figure 5.1: Rate remapping in the DG with spiking neurons. (A) Sample cells from MEC and LEC in the two environments. (B) Two sample DG cells exhibiting rate remapping. (C) PV correlation curve for the DG population compared with data from Leutgeb et al. (2007). The spatial response of 10,000 DG neurons was constructed from the spiking activity obtained from the simulation of individual gamma cycles in an integrate-and-fire network with delayed feedback inhibition over 5.2. RESULTS 77 real rat trajectories in open field protocols (Hafting et al., 2005) (Figure 5.1B). Temporal resolution was set to 0.1 ms. Samples from different trajectories were used when rate maps were compared. Overall excitatory input gain was selected in order to evoke gamma oscillation (37 ± 2 Hz). Relative MEC/LEC gain was set as 0.32 as in previous section. Most of the simulated DG neurons exhibited place fields with an average of 2.0 ± 0.8 per neuron (Figure 5.2A). Rate remapping was comparable to the real rat observation of the decorrelation of the population rate vector due morphing (Figure 5.1C). These results are in conformity with previous results obtained with a rate based model. In the simulated data the PV correlation curve is not affected by the direction of the morphing. This lack of hysteresis in the DG population has also been observed experimentally (Leutgeb et al., 2007). Following a similar method the response of 5.000 CA3 neurons was computed. The input originated in the DG was calculated from the rate maps built previously. LEC and MEC input were taken from the same population as the DG simulation. In a first simulation there were not considered recurrent synapses. Overall excitatory input gain was selected in order to evoke gamma oscillation (38 ± 1 Hz). Most of the simulated DG neurons exhibited place fields with an average of 2.0 ± 0.8 per neuron (Figure 5.2A). Most of the simulated CA3 neurons exhibited place fields with an average of 1.4 ± 0.3 per neuron (Figure 5.2B), confirming results from rate model (de Almeida et al., 2010). Next we verified whether the observed CA3 cells exhibit rate remapping. The PV correlation curve revealed that it does but not with the expected properties (Figure 5.3). For instance, the simulated curve exhibit significant difference between the correlation from the first arena to the second (1-2) and from the first arena to a subsequent recording to the same arena (1-1’). Moreover, the format of the curve was more similar to the one CHAPTER 5. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 78 THE CA3 Figure 5.2: Distribution of the number of place fields with the spiking model. Distribution for both DG and CA3 (no recurrents) is coherent with experimental findings (Leutgeb et al., 2007). observed in the DG. This is evidence that the recurrent input is essential for the changes in the PV correlation curve. Next we implemented the recurrent connections in the CA3 population. Following experimental procedure of Leutgeb et al. (2007) we trained the recurrents according to the population activity in the two extreme arena shapes using a hebbian-like function (de Almeida et al., 2007). The synaptic weight from cells i to j (W (i, j)) was defined based on the rate (r(x, y, s)) in every position (x, y) and arena shape (s). W (i, j) was calculated as follows: R(cell, r) = min(r, 10Hz); ∀(x,y,s) f1 (i, j) = ∀(x,y,s) f2 (i, j) = X r W (i, j) = R(i, r) ∗ R(j, r) 100 (5.2) R(i, r) ∗ (10 − R(j, r)) 100 (5.3) X r (5.1) f1 (i, j) 1.4 ∗ f1 (i, j) + 0.21 ∗ f2 (i, j) + 0.22 ∗ f1 (j, i) (5.4) 5.2. RESULTS 79 Figure 5.3: Rate remapping in the CA3 population without recurrents don’t explain experimental data. PV correlation curve for of the simulated CA3 population aligned to experimental CA3 (red ) and DG (blue) curves (Leutgeb et al., 2007). Once set the recurrent weights, we tested the effect of the synaptic strength in the PV correlation curve (Figure 5.4). The experimental curve could not be explained. Increase of recurrent strength caused an overall increase in PV correlation in all morphing stages. Although it was barely possible to obtain a flat correlation between 1-1’ and 1-2, in such situation the difference of correlation between 1-1’ and 1-7 (first and last shapes) was dramatically higher than the one observed by Leutgeb et al. (2007). A close look to the time dynamics of the neurons potentials shown that indeed the recurrents were able to cause pattern completion (Figure 5.5) but at a cost that the two extreme memories would be almost indistinguishable. CHAPTER 5. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 80 THE CA3 Figure 5.4: PV correlation curve for CA3 with recurrents batchtrained. Shown for multiple recurrent strengths (from black to light blue). Experimental curve, normalized to mean 1-1’ correlation in simulation, is shown in dotted red (Leutgeb et al., 2007). One important aspect of this simulation is that learning is applied before the trials. Thus, no hysteresis can be observed. Next we tested the effect of learning in rate remapping. Recurrent synaptic weight was set as before. Once started the task, it was computed a reference synaptic weight matrix (W temp (i, j)) based on the non-recurrent output of the CA3 population to the specific shape of the arena. Given a learning rate ψ defined in the range [0, 1], the synaptic weight matrix was updated as follow: W (i, j) = (1 − ψ) ∗ W (i, j) + ψ ∗ W temp (i, j) (5.5) The constant learning had strong impact in the 1-1’ correlation (Figure 5.2. RESULTS 81 Figure 5.5: Pattern completion by recurrent excitation. Trace of the potential of a neuron in a single gamma cycle for two different morphing stages. (top) When there is no recurrent input, the 10% morphing changes the input in a way that the cell cannot accumulate enough energy to release a spike. (bottom) When a recurrent input is present, the cell gets an extra amount of energy and spikes before the global inhibition is released. 5.6). Indeed, with ψ set to 10% the correlation in 1-1’ was much lower than in 1-2. This allowed a PV correlation curve similar to experimental data with ψ set to 5%. These results suggests that the experimental observation has been misinterpreted as if the output of the CA3 population in the morphing stage 2 is the same as the morphing stage 1. It seems that 1-1’ is not the same as 1-1 since the recurrent synapses has altered during morphing. This is an indication that memory is not static but it drifts with new experiences. Another indication of this phenomenon is that the correlation of two subsequent recordings is below the expected by sampling uncertainty (as observed by the 1-1’ correlation in Figure 5.4) which opens run for CHAPTER 5. MECHANISMS OF CONJUNCTIVE SELECTIVITY IN 82 THE CA3 Figure 5.6: Effect of LTP in the PV correlation curve. an extra source of variability in the signal. CHAPTER Mechanisms of hippocampal behavioral control As part of our effort to understand why place cells become conjunctive we studied how place cells can be used to support complex behavior. Our approach was to first relate the hippocampal mechanisms to behavior in a theoretical framework through computer simulations. In the subsequent study we used a brain-based robotic control architecture to experiment with robots in real-world tasks. Both studies present arguments of why conjunctive hippocampal representation is relevant to support complex task solving in theoretical and embodied grounds. 6.1 Theoretical study on behavior The first study is presented as the manuscript "Nonspatial selectivity of place cells supports quasi-optimal behavior in mixed spatial/nonspatial tasks" which is in preparation (Rennó-Costa & Verschure, 2012). The abstract reads: 84 6 6.1. THEORETICAL STUDY ON BEHAVIOR 85 The spatial selectiveness of hippocampal place cells is belief to be the computational basis of cognition for navigation in rodents. However, place cells are not only selective to position. Other nonspatial variables such as sensorial perception, behavioral context and emotional state also affect the firing of place cells. Based on the computational principles of convergent information integration, population feedback inhibition and sequencing, all observed in the medial temporal lobe, we used a computational model of the hippocampus and the entorhinal cortex to show that spatial selectivity is not sufficiently to solve mixed spatial/nonspatial tasks. Rather, the conjunctive representation of spatial and nonspatial information by the place cells supports quasi-optimal performance. From this principle is possible to predict behavioral performance given the complexity of any general task. These results attest for the behavioral importance of phenomena such as remapping and the joint codding of spatial and nonspatial information on place cells. 6.1.1 Introduction The observation of place cells is a major evidence of the existence of a cognitive map in the hippocampus (O’Keefe & Conway, 1978). Lesion of this brain region reveals behavioral impairment in spatial tasks (Morris et al., 1982), supporting that place cells play an important role in the cognition of navigation. This fact inspired computational models that based on place cells units to solve simple spatial tasks (Erdem & Hasselmo, 2012; Burgess et al., 1997). However, rodents are also able to perform spatial tasks that are dependent on nonspatial cues (Harrison et al., 2006). Indeed, lesions of the hippocampus impair object-location memory and the behavior in tasks that depend on objects or nonspa- 86 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL tial features (Gaskin et al., 2003; Mumby et al., 2002; Ennaceur et al., 1997). This raises the question of whether strict spatial selectivity is enough to support the behavior observed in rodents. That’s unlikely given that place cells are not only spatially selective. The granule cells of the dentate gyrus and the pyramidal cells of the CA3 region present place fields whose peak rates are modulated by nonspatial variables such as the shape and the color of the environment (Leutgeb et al., 2007). This modulation, known as rate remapping, is caused by the integration in the hippocampus of two separate spatial and nonspatial input channels from the entorhinal cortex (Hargreaves et al., 2005; Rennó-Costa et al., 2010a). While localized lesion of the medial entorhinal cortex, the source of spatial information (Sargolini et al., 2006), impairs exploratory behavior (Schenk et al., 1983), localized lesion of the lateral entorhinal cortex impairs object exploration but not pure navigation skills (Van Cauter et al., 2012). This suggests that the conjunctive representation of spatial and nonspatial information by the place cells is necessary for successful behavior in tasks that demand spatial and nonspatial reasoning. It remains unclear however the mechanism by which the nonspatial selectivity of place cells ultimately support exploratory behavior. Our approach to elucidate how the brain accomplishes exploratory behavior using conjunctive spatial/nonspatial representation was to use a virtual computational agent to perform two tasks in a virtual multiple Y-maze (Figure 6.1). In the navigation task the agent is rewarded whenever it performs a nonspatial action only available at a specific arm of the maze (Figure 6.1A). In the mixed task, the rewarded nonspatial action is only available after an intermediate nonspatial action is executed in another specific arm of the maze (Figure 6.1B). By analytically confronting the performance of a neural controller implementing place cells units with strictly spatial response profile (spatial controller) against another neural controller enhanced with conjunctive spatial/nonspatial selectivity (con- 6.1. THEORETICAL STUDY ON BEHAVIOR 87 A 1 2 3 4 R B 4 1 7 *8 decision point R rewarded action 3 2 § 5 6 R spatial action nonspatial action n optimal solution * avaiable after § Figure 6.1: Experimental protocol. Multiple Y-maze (lef t) shown with its graph representation (right). Decision points represented as vertexes, spatial actions as straight arrows and nonspatial actions as angular arrows. All affordances are shown. Optimal solution is in red. (A) Spatial task. Reward is delivered when the agent reaches a specific location and applies a nonspatial action. Illustrated as the task in which the rat has to find and eat a piece of cheese. (B) Mixed spatial/nonspatial task. Rewarded action (∗) is only available at the goal location after the agent applies a nonspatial action at a different location (§). Illustrated as the task in which the rat has to pull a button to release water in the fountain located elsewhere. junctive controller) it is possible to identify the functional role of the conjunctive coding in the place cells. The design of a neural controller that could, concurrently, be related to 88 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL the hippocampus functioning and be able to perform the maze tasks is facilitated by the fact that many of the underlying computational principles of the medial temporal lobe have already been identified: the interplay between gamma-modulated feedback inhibition (de Almeida et al., 2009a) and massive convergence of entorhinal input have shown how place cells emerge from grid cells (de Almeida et al., 2009b, 2010). The same principle accounts for rate remapping when considering nonspatial inputs from the lateral entorhinal cortex (Rennó-Costa et al., 2010a); the sequencing of episodic memory can be achieved in the entorhinal cortex (Hasselmo et al., 2000; Koene & Hasselmo, 2007) and stored and recalled in the DG-CA3 loop (Lisman et al., 2005; Lisman, 1999); pattern completion of hazy input in the CA3 (Leutgeb et al., 2007) can be explained by autoassociative synapses (Rolls, 2007; de Almeida et al., 2007). Some of these principles such as immediate and persistent memory sequencing and integrative memory formation have been identified in a specific neural model (Lisman, 2007). Called distributed adaptive control (DAC) (Verschure et al., 1992, 2003), it includes mechanisms for perceptual and behavioral learning which allow the self-organization of control rules from the interaction with the environment. DAC’s emerging features extrapolate the functioning of the hippocampus. This suits the need of closing the perceptual/behavioral loop in an embodied agent and at the same time allows scoping specifically the hippocampal related computation. Moreover, other relevant aspects might not be computed in the hippocampus and therefore can be considered available. For instance, self-location (Fyhn et al., 2004; Hafting et al., 2005) and context information (Burwell et al., 2004) is provided by the entorhinal cortex. Performed action information is provided by corollary discharge (Crapse & Sommer, 2008). Decision-making is performed by the medial frontal cortex, which integrates information from the hippocampus (Walton et al., 2002). DAC comprises three hierarchically coupled layers (Figure 6.2A). Se- 6.1. THEORETICAL STUDY ON BEHAVIOR 89 quential memories are formed at the ‘contextual’ layer from sensorimotor contingencies of associative perception and actions. These are learnt at the ‘adaptive’ layer from raw sensory input, motor primitives and predefined reflexes from the ‘reactive’ layer. In this study we bypassed the perceptive learning of the ‘adaptive’ control layer since this mechanism does not strongly depend on the hippocampus (Manns & Squire, 2001). This bypass facilitates observation and avoids performance artifacts due to behavioral feedback (Verschure et al., 2003). The defined associative perception included the position of the agent in the maze and whether the agent performed the intermediate nonspatial action or not. The associative action set includes the move from one location to another and the nonspatial actions. Affordances are available since they naturally emerge from the “adaptive” layer interactions with the environmental set (Duff et al., 2010), including point-to-point displacement in map-less navigation (Mathews et al., 2009). We assume salient events when the agent moves from one junction to another or after box-related actions. Salient events trigger the formation of perception/action couplets that are sequentially stored in the short-term memory (STM). STM is stored persistently in the long-term memory (LTM) when reward is delivered (Figure 6.2C). STM and LTM are, respectively, analog to the sequencing of memories in the entorhinal cortex (Koene & Hasselmo, 2007) and in the hippocampus (Lisman, 1999) (Figure 6.2B). Behavior is dominated by a specific LTM sequence when the STM match coefficient is the highest among all sequences and is above an activation threshold (Figure 6.2DE). If no LTM sequence is active, than the adaptive controller will trigger a random action among the affordances. As the perceptual input for the memory couplets we assume that precise self-localization information is available for computation, being provided through the outcome of a path integration mechanism in the medial entorhinal cortex (McNaughton et al., 2006). Although path integration 90 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL A Adaptive Perception Proprioception Action Reactive Sensory information Allostatic regulation Motor primitives External stimulus C D LTM CA3 DG LEC + STM Spatial Information = max STM Couplet Action Nonspatial Information MEC Cortex perception ABC éê ? E Recall LTM Reward Nonspatial Information CA1/SuB Response Behavioral Learning STM B Long-Term Memory Short-Term Memory Contextual Couplet Spatial Information Action LTM action CABCABCZ é êê êéêé é 1 0 0 2 0 0 3 0 max score next action: é Figure 6.2: The DAC architecture. (A) System overview and its major connections. The reactive layer relates statically sensory information and allostatic regulation with the motor primitives. The adaptive layer builds on top of the reactive layer with self-organized responsive units of perception, proprioception and actions. (B) Relevant connectivity in the medial entorhinal cortex. Dentate Gyrus (DG) and CA3 integrate the multimodal input from the lateral and media portions of the entorhinal cortex (LEC and MEC). Sequencing is obtained by the interconnectivity between CA3 and DG. Output is channeled back to the cortex through the CA1 and the Subiculum (SuB) (C) Schematic for behavioral learning (LTM acquisition) in the contextual layer. (D) Schematic for action recall. (E) Procedure to select next action. 6.1. THEORETICAL STUDY ON BEHAVIOR 91 is error prone in rats (Etienne et al., 1998), it can be corrected by the existence of stable environmental cues (Verschure et al., 2006). Also, sensory information or behavioral context is provided through the lateral entorhinal cortex. This area of the brain is highly innervated by sensory related areas such as the occipitotemporal cortex (McDonald & Mascagni, 1996) and the olfactory bulb (Carlsen et al., 1982). Moreover, behavioral context information might be provided by the prefrontal cortex through the entorhinal cortex (Hyman et al., 2005) and emotional context might be provided by the ventral striatum (Lansink et al., 2009). The computation of the memory units in the computational controller is analog to the integrative/competitive process that rise the formation of place fields in the dentate gyrus (de Almeida et al., 2009b) and in the CA3 (de Almeida et al., 2010). The two designated neural controllers differ by the information that is integrated in the formation of the memory couplets: the spatial controller only associates position to action while the conjunctive controller associate conjoinedly position and behavioral context. 6.1.2 Results The experimental protocol consisted of a learning phase and a performing phase. In the learning phase, the agent explored the maze by applying random actions within the acquired affordances. The phase was completed when LTM was full. In the performing phase behavior was dominated by the LTM sequences. To establish the upper and lower bounds of the performance we also considered a random controller with no LTM and an optimal controller that uses a shortest path algorithm (Dijkstra, 1959). The random controller succeeded in delivering a solution in every trial of each task. Time-out was set as 2 times the longest random solution of each experimental condition. Further trials that reach time-out were considered a failure. CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL A spatial place cells random Performance relative to the shortest path optimal 2 60 50 3 40 4 5 30 6 20 7 8 10 9 10 100 B Optimal memory length 92 101 102 Length of memory sequence 0 2 3 4 5 6 7 8 9 10 11 Mean optimal solution length 12 Figure 6.3: Spatial selectivity is sufficient for solving a spatial task. (A) Performance in the spatial task (optimal solution length 2.8±1.1 actions) as a function of the memory sequence length (median and interquartile range). (B) Estimated memory sequence length that leads to the best performance as a function of the mean length of the optimal solution of the maze. In the navigation task, the neural controller succeeded in taking the agent to the goal position in every trial and for every memory sequence length. Memory sequence length affected performance (Figure 6.3A). Performance peaked quasi-optimally with mid-lengthened memory sequences and deteriorated asymptotically to chance with short and long memory sequence lengths. This observation extended to different mazes complexities with maximum sequence length expanding exponentially with maze complexity (Figure 6.3B). In the mixed task, the controller based on purely spatial place cells failed to produce a solution in some task conditions with mid-lengthened memory sequences (Figure 6.4A). The initial position of the agent and the length of the optimal solution affected failure level. The controller based on conjunctive place cells succeeded in every trial and for every memory sequence length. Memory sequence length affected performance for both controllers (Figure 6.4B). Performance peaked quasi-optimally with mid- 6.1. THEORETICAL STUDY ON BEHAVIOR A Long optimal solution Short optimal solution 100% Conjuctive place cells Spatial place cells Random relative to the shortest path Successful trials optimal Performance 75% 50% 25% 0 10 1 10 Length of memory sequence C 2 3 4 5 6 7 8 9 10 0 10 2 10 Optimal memory length 0% B 93 1 10 2 10 Length of memory sequence 90 80 70 60 50 40 30 20 10 0 4 5 6 7 8 9 10 Mean optimal solution length 11 Figure 6.4: Conjunctive spatial/nonspatial selectivity is necessary for solving a mixed spatial/nonspatial task. (A) Percentage of successful trials of the spatial controller in the mixed task (optimal solution length 4.8±1.0 actions) and (B) performance relative to the shortest path in the mixed task as a function of memory sequence length (median and interquartile) for the half-shortest/-longest solutions. (C) Estimated memory sequence length that leads to the best performance as a function of of the mean length of the optimal solution of the maze. lengthened memory sequences and deteriorated asymptotically to chance with short and long memory sequence lengths. The conjunctive controller outperformed the spatial controller in the memory long sequence length range in which the spatial controller could produce valid solutions in more than 75% of the trials. This observation extended to different mazes complexities with maximum sequence length expanding exponentially with maze complexity (Figure 6.4C). One interesting aspect of the simulations in both tasks is that the size of memory sequences in rats is estimated as 7±2 memories (Lisman & 94 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL Idiart, 1995; Jensen & Lisman, 1996) which allow the use of the computational model to relate to the approximate performance of rats in mazes of specific complexity. The failure of the spatial controller to produce successful solutions with short memory sequences can be explained by the lack of behavioral context information in the recall of memories. In this situation, the sequences that lead to the rewarded action dominate the behavior whenever the agent is close to the ultimate goal location (Figure 6.5A). Evidence of this mechanism is that failure level is dependent on the initial position of the agent (Figure 6.4A). Conjunctive place cells solve this problem by disambiguating the position and the behavioral context so that behavior will only be dominated by the memories after the agent triggered the intermediate nonspatial action (Figure 6.5B). This mechanism is conform to experimental finding that place cells acquire directional selectivity when passing the same location in two different behavioral contexts of the same task (Navratilova et al., 2012). 6.1.3 Discussion These results evidence that neural computation performed by the hippocampus circuitry and based on conjunctive place cells is sufficient to solve mixed spatial/nonspatial tasks in a quasi-optimal fashion. Moreover, the hippocampus alone is not capable of solving the task relying solely in the spatial selectivity of place cells. Although other brain areas such as the medial prefrontal cortex (Erdem & Hasselmo, 2012) and the ventral striatum (Lansink et al., 2009) can possible provide the needed disambiguation tools for the spatial information of place cells, the theta synchronization between these areas and the hippocampus suggest a lower frequency information transfer than what is observed in the internal gamma modulated microcircuit responsible for sequencing, auto- 6.1. THEORETICAL STUDY ON BEHAVIOR A decision point n R rewarded action spatial action memory sequence § -3 § -2 R nonspatial action avaiable after § * § -3 -1 *0 95 -1 *0 -2 -1 *0 R -2 -3 R locked position B behavioral context 1 behavioral context 2 -3 -2 -1 0 -3 -2 -2 -3 R Figure 6.5: How conjunctive place cells solve the mixed spatial/nonspatial task. (A) 4-memory sequences of non-conjunctive place cells that cause misleading by attracting to action ∗ before action §being executed. The middle memory sequence is a special case in which the action §will never be accessible if the agent is at a locked position. (B) Conjunctive place cells solve it by establishing an independent graph for each behavioral context, causing that the agent will not be attracted to ∗ before executing §. 96 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL associative memory and phase-precession (Skaggs et al., 1996; Jensen & Lisman, 1996). This suggests that these areas provide important information that is integrated in the conjunctive representation of the place cells but that the computation itself is performed locally in the hippocampus. 6.1.4 Methods The multiple Y-mazes were built using a random Prim algorithm (Prim, 1957). Locations with 1 or 3 adjacent locations were considered as decision points (Figure 6.6). Nonspatial actions were set only on maze arms limiting to 3 the maximum number of actions available at a single position. The complexity of a specific maze was measure as the mean length of the minimal path to complete the task given a set of random initial positions. For the spatial controller, the memory couplet was set as <R,A> where R is the current agent location and A is an action. For the conjunctive controller, the memory couplet was set as <R,B,A> where B is the behavioral context. STM was set as FIFO queue of memory couplets of maximum size M. Memory couplets were pushed at the STM whenever an action was successfully applied. LTM was set as a list of memory couplet sequences of maximum size M with a maximum number of sequences N (N=32). Reward made the STM queue to be pushed at the LTM if LTM was not full. A persistent fitness value was associated to every memory couplet in LTM. Chaining was accomplished by transferring fitness value + 1 to the following memory when there is a hit. Values not transferred are zeroed. Action is selected as the one with highest fitness values among the affordances. If multiple actions have the same fitness, the action is selected randomly. If no action has fitness above threshold (T=1) than action is selected randomly among affordances. 6.2. ROBOT EXPERIMENTATION 97 Figure 6.6: Maze samples. With 10, 30, 100 and 129 decision points (topleft, top-right, bottom-left, bottom-right). Decision points in red and path in gray. One each task the agent explores randomly the maze until LTM is full. After, the agent preforms several trials relying on memory (500 trials per simulation). Each task condition (5 goals for each maze) was simulated 5 times. 6.2 Robot experimentation This work was published as the paper "Integrating neuroscience-based models towards an autonomous biomimetic synthetic forager", presented in the conference IEEE ROBIO 2011 in Phuket, Thailand (Rennó-Costa et al., 2011). As an extension of this robot setup we also developed a reactive controller designed to provide basic behavior (Appendix A) The 98 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL abstract reads: Foraging can be described as goal-oriented exploration for resources. It exemplifies how animals coordinate complex sensory and effector systems under varying environmental conditions. To emulate the foraging capabilities of natural systems is a major goal for robotics. Therefore, foraging is an excellent paradigm to benchmark novel autonomous control strategies. Here we describe the biomimetic control architecture of the Synthetic Forager (SF), an effort to integrate multiple biologically constrained models of specific perceptual and cognitive processes pertaining to foraging into one general autonomous robot controller. This proposal is built upon the well-established Distributed Adaptive Control (DAC) framework and brings together neuroscience-based models of decision-making, multi-modal sensory processing, localization and mapping and allostatic behavioral control. To show the potential of the SF model we used it to control a high-mobility wheeled robotic platform in three behavioral tasks similar to experimental protocols applied to rodents. We show that the robot can reliably perform cue detection, rule learning and goal-oriented navigation in open environments. We propose that this approach to robotics allows both the study of embodied neuroscience models and the transfer of brain based principles to robotic systems. 6.2.1 Introduction The central challenge of autonomous robotic technologies is the coordination of complex sensory and effector systems under varying task conditions. Successful systems have to account for rapidly changing demands from both the behaving agent itself and its environment. Additionally, all 6.2. ROBOT EXPERIMENTATION 99 this is to be accomplished under austere restrictions placed upon available computing resources and time, and upon the basis of incomplete and only partly reliable information. In nature, abundant examples of biological systems exist that fulfill all these requirements, sometimes in the most unpredictable and challenging ecosystems (Calhoun, 1963). Engineering solutions might profit if grounded in successful biological systems. An animal behavior of particular interest in this context is foraging, i.e. the ability of animals to optimally explore and environment and exploit its resources (Stephens et al., 2007). The concept of foraging is a step towards integrated situated behaving systems that further generalizes from biomimetic robotics examples such as legged dog (Song & Waldron, 1989) and fish robots (Chen & Zhu, 2005); chemical sensing (Mathews et al., 2009) to haptics using whiskers (Lepora et al., 2010); and evolutionary morphogenetic approaches (Jin & Meng, 2011) to name a few. Foraging is defined in behavioral ecology as the exploration for resources, usually motivated by deprivation, e.g., energetic and reproductive needs (Stephens et al., 2007). The concept encompasses goal-oriented behavior in complex environments where prior knowledge and action strategies must be matched to the novelty and hazards of a dynamic world and the varying requirements of the system itself. Successful foragers must simultaneously satisfy a wide range of constraints in varying spatial and temporal scales such as avoiding obstacles while maintaining an energyefficient trajectory back to a known feeding or nest site. In addition, due to natural selection and intense competition for resources, animals must perform in a near-optimal fashion. Indeed, this can be observed in a range of animal species from humans and monkeys to small insects (MacDonall et al., 2006; Davis, 1996). For example, it has been shown that rats choose their strategies based on the expected gain and its magnitude (Herrnstein, 1970). In a more specific case, rats have been shown to develop optimal adaptive foraging strategies with respect to travel 100 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL time, probability of food appearance and amount of food pellets in a radial arm maze, where food is delivered in different arms under varying conditions (Roberts, 1992). Although descriptive models exist for the behavioral aspects of foraging (Stephens et al., 2007; Giraldeau & Caraco, 2000), the underlying functional and neural organization have not yet been integrated into a unified theory. Given the complexity of foraging, such a theory must involve different levels of organization, from basic reflexes to social interaction. An inclusive understanding can only be reached if the individual aspects are combined in an integrated framework. Here we follow the Distributed Adaptive Control (DAC) architecture, a robot-based model of perception, cognition and behavior (Verschure et al., 1992). DAC has been successfully used in general tasks from mobile robotics (Mathews et al., 2009), interactive spaces (Eng et al., 2003) to synthetic music composition (Manzolli & Verschure, 2005). In addition, DAC has supported several biological findings such as behavioral feedback (Verschure et al., 2003), an integrated theory of classical conditioning (Verschure et al., 1992) and the principles of animal behavioral regulation (Sanchez-Fibla et al., 2010). In this study we present an integrated neutrally constrained model of foraging based on the DAC architecture. The Synthetic Forager (SF) is an integrated robotic mobile platform and biomimetic control architecture that realizes a foraging agent with a continuous duty cycle. SF is formulated against the benchmark of rodent foraging. SF integrates several neuroscience-based models including: decision-making and rule learning in the prefrontal cortex (Duff & Verschure, 2010), multi-modal sensory processing in the ventral visual stream (Wyss et al., 2003), localization and mapping in the hippocampus (Verschure et al., 2006) and allostatic behavioral control (Sanchez-Fibla et al., 2010). Here we describe how 6.2. ROBOT EXPERIMENTATION 101 the SF framework, the high-mobility platform to which it is applied, is used to control a wheeled robot is applied to three experimental tasks that capture protocols developed for studying foraging in rodents: (1) self-localization, (2) sequence learning and (3) resource localization. We demonstrate that the robot is able to succeed in these tasks suggesting that the integration of several specific biologically constrained models under the DAC framework will ultimately support optimal foraging behavior in robots. The synthetic forager framework SF is an integrated model of foraging based on the DAC architecture. It includes a series of sub models for the different requirements of foraging: allostatic control allows the management of different drives and associated behavior systems; the visual processing hierarchy provides for the transformation of high-dimensional input into a compact and usable representation; the egocentric and allocentric spatial processing hierarchy supports localization and mapping; the memory and decision-making systems ties the available perceptual information into causal relations to behavioral outcomes in the form of goal oriented sequences. These components are integrated in the DAC framework, rendering the SF control model. Distributed Adaptive Control (DAC) DAC (Verschure et al., 1992, 2003; Verschure & Althaus, 2003; Verschure et al., 2006; Duff et al., 2010) is a robot-based neuronal model of perception, cognition and behavior that is a standard in the domains of new artificial intelligence and behavior-based robotics. It is constraint by biology and fully grounded since it autonomously generates representations of its primary sensory input. This facilitates a direct comparison to bi- 102 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL ological systems and the thorough investigation of complex phenomena, such as foraging, and the generation of internal states that are related to sensory, motor, cognitive or motivational states and the influence of the environment and behavior on this process (Duff et al., 2011). DAC is organized around three tightly coupled layers: reactive, adaptive and contextual. In the reactive layer, a prewired set of reflexes enables the behaving system to exhibit simple reactive behaviors. The adaptive layer uses the cues provided by the reactive layer to acquire sensory representations and their associated behaviors. This provides the mechanisms for the adaptive grouping of sensory events and the reshaping of responses, as observed in classical conditioning (Duff et al., 2010). The contextual layer acquires, retains, and expresses sequential sensory-motor contingencies, e.g. perception-action tuples (Duff et al., 2011), provided by the adaptive control layer, using mechanism for short- and long-term memory. This mechanism describes goal-oriented learning as in operant conditioning. The dynamics of these memory structures during a foraging task are equivalent to a Bayesian description of foraging (Verschure & Althaus, 2003). Allostatic control The allostatic control system of the reactive layer deals with specific drives, e.g., hunger and safety, and modulates behavior by establishing specific reactive goals. In SF, the allostatic system is composed of actively updated state variables that are connected to reactive behaviors and/or are available to the decision-making system. For example, state variables might be described as internal needs such as “hunger” which would initiate and sustain a “seek for food” behavior. The allostatic model regulates multiple systems that collectively generate complex task dependent behavior. 6.2. ROBOT EXPERIMENTATION 103 Visual processing hierarchy In SF, vision is the main distal sensory input. The aim of the visual processing hierarchy is to provide a robust representation of salient elements of the world that support navigation and goal oriented behavior. In SF we use the SIFT algorithm (Lowe, 1999). Spatial processing hierarchy Spatial localization and mapping is a major element of foraging. Some important notions as home/nest and resource disposal depend on the abilities of acquiring the knowledge of where the system itself is and how this information spatially relates to relevant sites. In the DAC architecture, memories are perception-action tuples formed in the adaptive layer. There is functional evidence for this spatial/non-spatial association in the hippocampus (Lisman, 2007), which is strengthened by the fact that the spatial code is modulated by manipulations of non-spatial variables, in a phenomenon called rate remapping (Leutgeb et al., 2007). This aspect of the the adaptive layer is based on a recent model of the hippocampus where place cells are formed from the principles of convergent signal accumulation and population derived feedback inhibition (de Almeida et al., 2009a), which explains the formation of place cells from grid cells (de Almeida et al., 2009b) and rate remapping (RennóCosta et al., 2010a). In the model, the hippocampus receives as input: the response of a path integration system, which is updated by odometry and regulated by place cell activity (Guanella & Verschure, 2006); the output of the visual processing system, which provides landmark information (Verschure et al., 2006); and motivational information provided by the allostatic control (Sanchez-Fibla et al., 2010). The output of this system provides input to the decision making and memory system in spa- 104 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL tial tasks. Memory and decision making In DAC, memory is organized in the contextual layer by means of sequencing, e.g., active memories trigger subsequent memories through lateral interaction (Marcos et al., 2010). Memory sequences are built from the sequential occurrence of single memories, i.e. sensory-motor tuplets, and are stored in a short-term memory buffer, thus sequencing depends on their occurrence in time. Whenever a goal relevant event is detected - when a cue is detected, the robot arrives home or food is found, among others – the full content of the short-term memory is stored in the long-term memory. Recall is based on the matching of the elements of these stored sequences to ongoing sensory events. When a memory element is active and contributes to action it will bias the next memory element in its sequence increasing the probability that it contributes to action in the future. Robots For the experiments described here we used two wheeled robots designed and constructed by Robosoft (Bidart, France). The outdoor unit is a prototype in development (Figure 6.7 left) whereas the indoor unit is a standard robuLab 10 (Figure 6.7 right). Each unit is capable of doing on-spot turning and is equipped with infrared and ultra-sound proximity sensors covering a range of 360o . Both robots are equipped with a color CCD camera (Imaging Source, Bremen, Germany) mounted on a pan-tilt unit (Direct Perception, Burlingame, USA). Both robots include an embedded computer system running Linux Ubuntu. Control software was implemented as a stand-alone application in ANSI C++. 6.2. ROBOT EXPERIMENTATION 105 Figure 6.7: SF Robots. (left) Outdoor and (right) indoor units with 1.1 x 0.6 m and 0.6 x 0.6 m respectively. Both equipped with embedded computation, proximity sensors and color camera mounted on a pan-tilt unit. 6.2.2 Results To test the ability of DAC to generate foraging we applied the SF system to three tasks critically involved in foraging: (A) in the self-localization task we seek the ability of creating a robust internal representation of the position of the robot; (B) in the sequence-learning task the objective is to learn relations among causal events that can be expressed in rule based goal oriented behavioral sequences; (C) in the resource-localization task the robot is expected to first associate the position of a resource to a variable visual cue and second to be able to reach the acquired target location. For each task we predefined perception primitives, action sets and allostatic values or drives. The connection between perception, action and allostatic values of the reactive layer is predefined in the form of causal rules (cause→consequence). In the adaptive layer, complex perceptual states are formed, as for example in position specific memories and these are associated with both allostatic values and actions. In the contextual 106 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL Figure 6.8: Outdoors arena. (A) Overview of the operative area. (B) Snapshots of the robot (90o between each other). layer, there are predefined events - such as achieving specific goal states - that trigger the formation of contextual memories in the form of sequences. Self-localization task The self-localization task is designed to test the adaptive formation of position memories. It is a protocol in which the robot moves semi randomly in an open field arena with the objective of developing a neural representation of its position. It is comparable to the standard open field task used in the description of the, so called, place cells in which the rat explores the arena without explicit goals (O’Keefe & Dostrovsky, 1971). For the robot version, it was performed outdoors in a campus square surrounded by buildings of different shapes and colors (Figure 6.8A). The operational area was limited to a rectangle of 8.8 x 6.4 meters. Tracking applied in the analysis was performed by referring to positioning marks placed on the floor with a resolution of 0.8 x 0.8 meters. 6.2. ROBOT EXPERIMENTATION 107 The reactive behaviors in this task were the following three reactive actions: (A1.1) “random walk”, in which the robot performs a small random rotation (−90o to +90o ) followed by a forward movement (0.8 meters); (A1.2) “orienting”, in which the robot takes snapshots from −135o to +135o with 45o steps (Figure 6.8B); and (A1.3) “homing” in which the robot is manually oriented to the center of the arena using joystick control. We defined three allostatic drives: (D1.1) “exploration”, (D1.2) “novelty” and (D1.3) “out of arena”. D1.1 is always active while D1.3 is set externally using a joystick whenever the robot is located outside of the arena. No reactive perceptual states were defined. The reactive rules are: D1.1 + not D1.2 → A1.1, not a novel place leads to exploration; A1.1 → D1.2, exploratory action leads to a novel place; D1.1 + D1.2 → A1.2, a novel place leads to cue gathering; A1.2 → not D1.2, cue gathering reduces novelty; D1.3 → A1.3, escape of the arena leads to control by the user. The visual processing system was based on SIFT. In each picture we computed a collection of local salient points using the SIFT algorithm (Lowe, 1999). Snapshots taken at each corner with an angle step of 90o covering a 360o total view served as a reference. Whenever A1.2 was executed, the orientation of the robot was measured by comparing 3 pictures with 90o angle step to the all reference pictures sequential combinations. The robot orientation was recalled as the one with the combination of reference pictures that maximizes the number of matching salient points. Once the orientation was set, the mean change in scale from the corresponding images to the 3 reference pictures sharing the same alignment define the output of the visual layer (V O). The V O comprises 16 cells (4 corners x 4 orientations). In a map like representation, the output of the visual layer shows strong position specific modulation (Figure 6.9A). 108 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL The memory layer activity uses the visual layer output as input. Computation is performed by a model of the input-output transformations of the dentate gyrus (de Almeida et al., 2009b; Rennó-Costa et al., 2010a). The input M Ii of a memory cell i at position r is defined as: M Ii (r) = 16 X e−2∗|V Oj (r)−αij (r)| (6.1) j=1 Where for each cell i in the memory layer we defined a preferred value αij for the visual cell j. αij was set randomly in the range [0, 2]. The output M Oi of the memory cell i is a result of a competition process in the form of a E%-MAX winner-take-all mechanism (de Almeida et al., 2009b,a, 2010; Rennó-Costa et al., 2010a): M Oi (r) = max (0|M Ii (r) − (max1≤k≤1000 M Ik (r)) ∗ .9) (6.2) Through this process the memory population exhibits place cell like responses, e.g. the activity of the cells is limited to a certain region of the arena (Figure 6.9B), in contrast to typical non-spatially specific response found for the visual layer. In order to access the precision of the position information, standard Bayesian reconstruction (Guanella & Verschure, 2007) was applied to the readout of the self-localization data revealing a mean normalized error of 1.1 meters in both layers (equivalent to 2.4% of the arena). Moreover, the population vector autocorrelation analysis (methods in Leutgeb et al. 6.2. ROBOT EXPERIMENTATION 109 Figure 6.9: Neural representation of space. Rate maps of (A) three visual layer cells and (B) three memory layer cells. X- and Y-axis represent position and brightness the unit activity. (C) Population vector selfcorrelation for both layers. (D) Boxplot of spatial-info score in bits/spike. (2007)) showed a more stressed incremental disparity in the population activity as a function of the distance for the memory layer (Figure 6.9C). This is an expected property of pattern separation in the dentate gyrus (Leutgeb et al., 2007). Furthermore, the activity of single memory units carries more information about the position of the animal than single visual cells (Figure 6.9D, methods in Skaggs et al. (1993)). These results together show that the visual model is successfully capturing the position of the robot and that the memory model is capable of representing this information in a compact and differentiable way supporting a symbolic- 110 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL like computation in a fully grounded manner. Sequence-learning task The sequence-learning task is a protocol in which the robot has to learn a color sequence. In the task, the robot has to move to a specific position indicated by a red light (Figure 6.10A) where two colors, one in each side of the robot, are displayed on the floor (Figure 6.10B). The robot has to gather the color cues and after point to the correct color. If it points to the correct side two new colors are presented, following the sequence (Figure 6.10C). If the robot makes the wrong decision the floor blinks in red and the robot has to return home to restart the procedure. This procedure is inspired by sequential water tank visual discrimination task (Aggleton et al., 2010). The cognitive system was programed with five actions: (A2.1) go forward; (A2.2) go backwards for 3 meters; (A2.3) visual orienting (set pan tilt −90o , gather color A, set pan +90o gather color B); (A2.4) point left; (A2.5) point right; Four drive states were defined: (D2.1) “sequence on”; (D2.2) “at position”; (D2.3) “memory full”; (D2.4) “wait answer”; A further reactive perception event was defined (P2.1) when a red color patch was detected on the floor and a further internal state (P2.2) when a correct selection was made. The reactive rules defined were: D2.1 + not D2.2 → A2.1, if sequence is on and not in position, go forward; D2.1 + not D2.2 + P2.1 → D2.3, when red is detected, it is in position; D2.1 + D2.2 + not D2.3 → A2.3, if no memory check for options; A2.3 → D2.3, if color checked then memory is full; A2.4 or A2.5 → D2.4, if selection made, wait answer; D2.4 + P2.4 → not D2.1 + A2.2, if red is detected when a sequence is completed with a wrong answer, go home; D2.4 + (delayed) not P2.4 → not D2.3 + P2.2, if no red is detected, continue the sequence. To realize this task, the vision system detected saturation 6.2. ROBOT EXPERIMENTATION 111 Figure 6.10: Sequence-learning task. (A) Robot goes forward until the red mark is detected. (B) Two color options are presented to the robot. (C) If the correct color is selected then two new options are presented. and hue divided in 12 hue bands. The perceptual segment of memory units stored hue value. Memory sequences were recorded whenever P2.2 was triggered. Through perceptual matching, stored long-term memory elements can be activated and the location of the next cue and the direction in which to move the camera is selected according to the predicted hue. If no memory is activated then a color location is selected randomly. The robot could correctly acquire and perform the whole sequence (depth = 4) in all runs (n=10). Starting with an empty memory it took a mean of 2.8 trials to accomplish the whole sequence. The robot performed the task successfully for a public demonstration performed in the XIM space (Bernardet et al., 2007). 112 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL Figure 6.11: Resource-localization task. (A) Experimental space (20 x 15 meters) with the arena marked on the floor. (B) Cue gathering at home. (C) Kidnap procedure. Robot is taken from one position to another and successfully finds the cued rewarded location. Resource-localization task The resource-localization task is a behaviorally active protocol in which the robot has to recognize a conditioned stimulus presented at his home/nest and associate it to the expected position of the reward (Figure 6.11B). The reward location is assigned to a specific position deterministically according to the cue. The task is conceptually similar to an experimental protocol used to study decision-making in rodents. In this T-maze task rata could either choose to climb a barrier to obtain a high reward in one arm or could obtain a small reward in the other with no barrier present (Walton et al., 2002). The robot T-maze task was developed indoors (Figure 6.11A). The operational area was restricted to a 4.8 meters wide square in the middle of 6.2. ROBOT EXPERIMENTATION 113 the experimental room. Cues were placed on all walls to provide distal landmark references that can be captured by the visual system. One wall of the arena was marked as home while the three others were marked as target/food disposal localities. The reactive layer of the robot was predefined with the drives: (D3.1) “at home”; (D3.2) “food available”; (D3.3) “at food position”; (D3.4) “outside of the arena”. The specific actions were defined as: (A3.1) “orient towards cue”, set pan-tilt up and rotate to search for a visual cue; (A3.2) “go to specific position”; (A3.3) “check current position”, which is based on the self-localization procedure defined previously; (A3.4) “go home”; The rules were: D3.2 → A3.2, whenever there is food, go towards it; not D3.2 + not D3.1 → A3.4, if no food is available then go home; not D3.2 + D3.1 → A3.1, if no food available and at home, seek for a cue; D3.3 → not D3.2, whenever food is found, consume it; D3.4 → A3.3, leaving the arena initiates a check for the current position. Cue detection is provided by the visual system by segmenting the cue carrier (fluorescent yellow card) and applying matching on the salient points obtained with SIFT. The specific cue is selected based on the class with the highest scale and rotation correlation among all other candidates for four sequential frames. The goal-oriented memory, is represented by a sensorimotor tuple: <cue>/<position>. Memory is consolidated in LTM whenever the robot finds a novel rewards site without using the long-term memory. When a cue is detected but no memory unit is associated with it, the robot will do a random search following the procedure described in the self-localization task to establish the relationship of the novel cue with a specific position. The robot successfully performed the task in a live demo for an audience of 20 people under three conditions: (1) new cue / food location; (2) 114 CHAPTER 6. MECHANISMS OF HIPPOCAMPAL BEHAVIORAL CONTROL known cue / food location; and (3) know cue / food location with kidnap, e.g., the robot is stopped and transferred to another place during the execution of the task (Figure 6.11C). When including all performance sessions (n=8): cue detection was successful in 91% of the trials (n=78); the robot always could find the correct position, relying only on path integration in 60% of the trials and requiring one or more position reconstruction procedures using visual information in the other 40% trials. 6.2.3 Conclusions Here we presented the SF control architecture, based on DAC, which solves behavioral tasks similar to those applied in the study of animal behavior. We use foraging as a main benchmark since it comprises a range of motivational, perceptual, cognitive and behavioral systems that well summarize the main aims of autonomous robotics while representing a major area in the study of animal behavior. The current implementation of SF shows how the multi-layer organization of actions, perception and drives combined with mechanisms of learning and memory can be used to express foraging behavior. Moreover, our approach not only allows the performance of behavioral tasks using robots but also provides methods to validate specific neuroscience-based models. As an example, the SF system for localization and mapping provides a biologically based system comparable to the SLAM paradigm (Leonard & Durrant-Whyte, 1991). As a drawback, the current model still depends on dedicated task definitions for perception, action and drives. Although some of these systems will rely on strong genetic pre-specification the role of ontogenesis in the generation of these systems in these systems will be expanded in future instantiations of the SF-DAC model. Further steps include the development of a basic set of actions, perception and drives that can generalize 6.2. ROBOT EXPERIMENTATION 115 to a larger range of tasks. Such progress is already considered in recent DAC related work (Duff et al., 2010) and might support what could be defined as a general synthetic foraging agent. CHAPTER Conclusion This dissertation is an effort towards a definition of The Hippocampus Code, the underlying computational structure underlying hippocampal function. In this direction, the set of studies herein presented is attentive to the ability of hippocampal neurons to be conjunctively selective to spatial and nonspatial aspects of memory. In a first stage (Chapter 4 and 5), we presented a systematic explanation of how the neural network in the hippocampus evokes such conjunctive selectivity. Further (Chapter 6), we used a system-wide robotic control architecture to determine, both theoretically and experimentally, why conjunctive selectivity is relevant for behavior and efficient task-solving. Hence, our major contributions are the demonstrations of how and why hippocampal cells become conjunctive, a characteristic that encompasses both memory and space theories for the hippocampus. The implications of our findings span throughout a wide range of scientific domains, from biology and neuroscience, behavioral psychology, cognitive and computer science and robotics. This is illustrated by the di116 7 117 versity of symposia and conferences in which some parts of this work have been included. Some of these events are: the Society for Neuroscience meetings in San Diego and New Orleans; the congress of biomimetic robotics (ROBIO) organized by IEEE in Phuket, Thailand; the Living Machines robotics conference in Barcelona; the Federation of European Neurosciences (FENS) meetings in Amsterdam (in which a grant was guaranteed by the Spanish Society for Neuroscience) and Barcelona; and the Cognitive Systems (CogSys) conference sponsored by the EUCog network in Vienna. In what regards to biology and neuroscience we advanced the knowledge of the rate remapping effect exhibited by the hippocampal place cells. We posed specific predictions about the importance of brain areas such as the lateral entorhinal cortex and provided new interpretations of neurophysiological data concerning attractor dynamics, pattern completion and long term potentiation in the hippocampus. A prove of relevance of the work is the fact that it was published in a high impact peer reviewed journal in the neuroscience field (Rennó-Costa et al., 2010a), which included an introductory article by Dupret et al. (2010). As a natural follow up of these studies remains some open questions regarding the time dynamics of the rate remapping effect. For instance, how would phase precession affect rate remapping? How does the dynamics of theta cycling changes the internal competition? How is the CA1 interaction with the rate remapping mechanism? The tools we used on this work allow the study of other relevant mechanisms of the hippocampus. The results of the behavioral studies also have they share of relevance in these fields since it allows a causal link between physiology and behavior. Yet, to accomplish higher impact it requires some refinement 118 CHAPTER 7. CONCLUSION on the methods to allow a better match to existing behavioral data or the realization of behavioral studies following the proposed experimental protocol. Still, the possibilities are very auspicious given that nowadays causality between behavior and physiology of the hippocampus is only accomplished indirectly by means of lesion studies. Far beyond, the ability of using robots will very likely be the ultimate benchmark for biology models by allowing the closed cycle including perception, cognition and behavior in realistic and natural environments. A previous study using the same framework and a very simplistic robotic setup already allowed the identification of the novel biological principle of environmental feedback (Verschure 2003). With the advance of computational models and the use of integration architectures, as the one presented in Chapter 6, the robots might soon become an essential tool to study the hippocampus and the formation of memory by grounding in behavior. Moreover, this method can potentially affect the study of behavior itself by fostering the identification of its underlying mechanisms. It is important to phrase that the establishment of this scenario requires detailed modeling of many parts of the brain and nervous systems. The current state-of-art already supports simple biomimetic behavior like the one presented in Chapter 6, but there is high demand for complex and detailed computational models of the brain. Not limited to biology, this work also has implications in applied sciences such as robotics and automation, in computer science and in cognitive science. The identification of computational mechanisms in biology might allow the development of biomimetic solutions for real-world engineering problems. For example, rodents excel in foraging tasks and suit as a reference model for robots to be designed for tasks that require the same set of skills such as resource-finding in unfriendly environments (e.g., underwater oil exploration) and automated rescue missions in harsh con- 119 ditions (e.g., earthquake and radiation sites). Furthermore, our findings might also allow the identification of novel algorithms, specially taken in consideration the advance of neuromorphic computational architectures. For instance, models of the hippocampus will be able to provide compact memory sub-systems for non-symbolic architectures. Although much of these scenarios are already tangible, there’s still a long road to be trailed before being effective. It rests no doubt however that the advance of many of the listed fields will be supported by interdisciplinary studies like this dissertation. APPENDIX Internal drive regulation of sensorimotor reflexes This appendix chapter reproduces the paper "Internal drive regulation of sensorimotor reflexes in the control of a catering assistant autonomous robot" published in the proceedings of the Living Machines conference (Rennó-Costa et al., 2012c). The abstract reads: We present an autonomous waiter robot control system based on the reactive layer of the Distributed Adaptive Control (DAC) architecture. The waiterbot has to explore the space where catering is set and invite the guests to serve themselves with chocolate or candies. The control model is taking advantage of DAC’s allostatic control system that allows the selection of actions through the modulation of drive states. In the robot’s control system two independent behavioral loops are implemented serving specific goals: a navigation system to explore the space and a gazing behavior that invites human users to serve themselves. By approaching and gazing 120 A A.1. INTRODUCTION 121 at a potential consumer the robot performs its serving behavior. The system was tested in a simulated environ-ment and during a public event where it successfully delivered its wares. From the observed interactions the effect of drive based selfregulated action in living machines is discussed. A.1 Introduction Many day-by-day human tasks require the ability to interact with others. That is the case of serving chocolate and candies in a social event. The waiter has to navigate through the hall, approach the guests and invite them to try what is on his plate. An optimal waiter would be able to map the space and program the visit to every guest remembering the time constraints involved. Those are not simple tasks for an autonomous robot. In this paper, we propose a purely reactive controller aimed to allow robots to assist catering services. Although it does not envisage a performance comparable to a human waiter, we suggest that the complexity added to the agent’s behavior by the internal regulation of the sensorimotor reflexes can establish a non-verbal communication channel and create a rich and effective serving experience to the user. The approach taken is to communicate with the guest through gaze (Knapp & Hall, 2009) and smooth reduction of interpersonal distance (Lawson, 2001; Inderbitzin et al., 2009). This is accomplished by the control of a camera mounted on a pan-tilt unit and the navigation of a mobile robot. The theory we pursue is that complex behavior, such as effective serving of food might emerge from simple and limited constructions without relying on complex processes such as representations, memory or inference skills (Braitenberg, 1984). The apparent complexity of behavior over time is a reflection of the complexity of the environment in which the agent finds itself (Simon, 1969). By allowing basic behaviors of gaze and motion to be coordinated through the interaction with the en- 122 APPENDIX A. INTERNAL DRIVE REGULATION OF SENSORIMOTOR REFLEXES vironment we aim to achieve emergent behavioral regularities: designing for emergence. Our approach is essentially different from the traditional top-down robot design methodology in which the environment and the possible interactions are parameterized and behavior follows specific and declarative predefined schemes (Pfeifer & Verschure, 1994; Pfeifer & Bongard, 2006). Designing for emergence might appear more of an art than a science but we base our methods on the fact that the synergy between perception and behavior is mediated by the environment in controllable ways (Verschure et al., 2003). This supports a bottom-up robot design methodology grounded on a more generic and also simple control system. The complexity of the interaction is expected to emerge naturally from the immersion in the environment and from the contact with other agents such as animals (Lund, 1997), humans (Eng et al., 2003) or other robots (Asama et al., 1994). Robots that can establish a closed loop in the sensorimotor interaction with the environment and generate emergent complex behavior have been called living machines (Hasslacher & Tilden, 1995), which established a significant field inside the autonomous robotics community (Bekey, 2005). The objective of the proposed control system is to implement a living machine with behavioral complexity that matches the environmental requirements and human expectations of the catering task, without the need for explicitly and centrally declaring the task in the robot control system. The main source of behavioral complexity in the control system we propose is the dynamic regulation of the sensorimotor loops through internal drives. Drives are defined as internal states that are defined by the fundamental homeostatic needs of the agent and that adjust the link between perception and action. Drives are set dynamically by the activity of the system and may follow different time scales than the time course of a A.1. INTRODUCTION 123 motor action or the sensorimotor cycle. These two properties together allow a repertoire of multiple responses to the same stimuli with a varying time frame that might not be perceived as a change in perceptual reaction (Simons & Levin, 1998). The implementation of the control system is based on the Distributed Adap-tive Control (DAC) architecture (Verschure et al., 1992, 2003; Verschure & Althaus, 2003; Duff et al., 2010). DAC is a sensory-action closed loop embodied control system based on the theories of conditioning and grounded on its neurophysiological constraints. Its vertical multi-layered scheme allows the acquisition of internal representations of conditional stimuli with crescent complexity sup-porting adaptive environmentalmediated behavior (Verschure et al., 2003). The implementation of internal drive in DAC’s reactive layer allows the low-level regulation of vital needs in an allostatic control framework (Sanchez-Fibla et al., 2010) and supports goal-oriented behavior by acquiring higher cognitive representations such as rules and plans (Duff et al., 2011). Although DAC provides the tools for learning and adaptation, we limit the robot’s controller to an implementation of the reactive layer. The reactive layer offers a basic repertoire of actions, sensations, reflexes and drives. Through this basic control system only non-adaptive behavior is generated since no persistent memory of any kind is implemented. The emerging sensorimotor dynamics might ultimately form the basis for the acquisition of internal representations and adaptive behavior (Duff et al., 2010), so this structure might work as a base for adaptive behaving system working in the same environmental context. Another important aspect is that DAC has already been used in the control of living machines that interact with humans. It is the case of the Ada interactive space (Eng et al., 2003). Ada is a room that interacts 124 APPENDIX A. INTERNAL DRIVE REGULATION OF SENSORIMOTOR REFLEXES with the user using light and dynamically composed music (Manzolli & Verschure, 2005). A key element for evoking the sense of interaction in humans is the ability to generate complex and unpredictable behavior, which does not look (or sound) like a pure reactive response (Michaud et al., 2005). In this respect, as a benchmark the control system, it was used in the control of an unmanned mobile vehicle in a real catering situation during the Future and Emergent Technologies (FET’11) meeting in Warsaw. In the following sections we describe the control architecture, the results of computer simulations and provide a brief report on the public demonstration of the system. We further discuss the effect of drive regulation and the limitations imposed by the use of purely reactive systems. A.2 A.2.1 The control architecture The hardware The robotic platform used to implement the living machine is a 50 kg unmanned mo-bile vehicle (124 x 80 x 67 cm) with 6 wheels and an articulated body with 3 subsections (Figure A.1). Its sensory system is composed by an array of 16 ultrasound and 16 infrared proximity sensors and a Firewire color camera mounted over a pan-tilt unit in the front body subsection. A notebook (MacBook Pro running custom application under Linux Ubuntu) that is docked in top of the middle body subsection performs computer processing. The load to be distributed is placed in the back body subsection. Robosoft (Bidart, France) developed the platform in cooperation with Universitat Pompeu Fabra (Barcelona, Spain). As an external interface, an iPad with a custom application allowed the visualization of all variables in real time, the override of the navigation loop and emergency locking. The simulation environment used to opti- A.2. THE CONTROL ARCHITECTURE 125 Figure A.1: Robot overview. The color camera is mounted over the pantilt unit in the front segment of the robot. The control CPU is placed over the middle segment. The load is placed over the rear segment. The proximity sensors are placed all around the robot, but only the ones in the front and in the back are used for the experiment. Robot produced by Robosoft. mize the parameters of the reactive controller (Sanchez-Fibla et al., 2010) allows a control that is similar to that of the real robot. One of the main features of the simulator is the possibility to customize different virtual environments and realize experiments in computational time rather than real time. A.2.2 The autonomous control system The autonomous control system is divided in two main sensorimotor loops: the vision loop and the navigation loop. This is an approximation of the subcortical loops formed by the basal ganglia and brainstem sensorimotor structures comprise sensing, internal motivational states and action (McHaffie et al., 2005). While the navigation loop allows the robot to explore the space searching for unattended guests, the vision loop will permit “eye contact” through the gaze system, showing to the user the “intention” of the robot to serve that specific person. 126 APPENDIX A. INTERNAL DRIVE REGULATION OF SENSORIMOTOR REFLEXES Figure A.2: From face perception to gaze action: the perception/action reflex in the visuomotor loop (A) Illustration of the visual field of the robot and (B) the associated salience map emitted by the face detection and salience map system. (C) Through a competitive process the most salient point in the visual field is selected. If the most salient point is active in a zone associated with a gaze action, a saccade is activated. In this specific case, the action “Pan Left” is triggered, moving the detected face to the center of the visual field. (D) Illustration of the two possible action types ("tilt" and "pan”) in relation to the gaze direction. Both sensorimotor loops conform to structures with the same components: perception, action, drive and reflex-es. Perception and action are the sensor/actuator interfaces to the environment. Drive is the internal state of the agent. Reflexes are the hardwired connections among these components. Each control loop is independent and no information or signals are inter-changed. Moreover, they differ with respect to the types of perception, action, drive and the specific organization of the reflexes. A.2. THE CONTROL ARCHITECTURE 127 The vision loop receives sensory input from a color camera and acts through the motorized pan-tilt unit. The expected behavior is that the robot tracks faces using the gaze system. Faces are detected by a cascade of boosted classifiers working with Haar-like features and trained with face samples (Lienhart et al., 2003). The output of the visual processing is a salience map that is aligned to the Cartesian representation of the retinotopic input (Figure A.2AB) and can be seen as an approximation of the sensori-motor mappings found in the superior colliculus (Song et al., 2010; Gandhi & Katnani, 2011). We interpret the active control of the gaze unit in terms of a cerebellum-like saccadic control system acquired through conditioning-like learning mechanisms (Schweighofer et al., 1996; Hofstötter et al., 2002; Blazquez et al., 2003). From this salience map it is possible to determine whether there are relevant eye movement targets, i.e. faces, in specific areas of the visual field. An attention mechanism based on competition and predictive anticipation from the recent response history is used to select one single salient face in the visual field. The algorithm consists of searching for a peak in the neighboring area of the last salient point. If no peak is found (highest value in the neighborhood is below the salience threshold, Ω), it searches for the highest peak in the whole visual field. The neighborhood is defined as the predicted area of the highest likeness of finding new salient point given recent history, or an anticipatory gate (Mathews et al., 2009). The algorithm that define the salient points follows: neighbor ← computeN eighbor(currentSalientP oint) if maxSalience(neighbor) > salienceT hreshold then currentSalientP oint ← maxSalience(neighbor) else if maxSalience(visualF ield) > salienceT hreshold then currentSalientP oint ← maxSalience(visualF ield) else currentSalientP oint ← center(visualF ield) end if 128 APPENDIX A. INTERNAL DRIVE REGULATION OF SENSORIMOTOR REFLEXES end if Reflexes were set connecting the face detection output to the movements of the pan-tilt unit (Figure A.2CD). The visual field was divided in five horizontal and five vertical zones. To each zone two reflexes are assigned, one horizontal and another vertical. The movement triggered by a certain reflex is capable of centering the camera in the active zones. In the vision sensorimotor loop, the drive variable is analog to the concept of “curiosity” or “novelty” (Figure A.3). Higher levels of curiosity make the gaze system search for new faces, whilst lower levels of curiosity make the gaze system stick to a certain face or remain still. The level of motor action regulates the curiosity level. This is an indirect measure of the variability of the visual input, which allows an environmental mediation of the drive regulation. This is an visual analog of exploration behavior (Sanchez-Fibla et al., 2010). Figure A.3: Visual loop organization. Perception excites action. Action inhibits the drive. The drive inhibits perception. Action and drive have spontaneous activity. Drives regulates the system by allowing action spontaneous activity to take place when few actions are triggered. The perceptual activity is modulated by the curiosity drive through an inhibitory reflex (Equation A.1). The curiosity drive is set accordingly to A.2. THE CONTROL ARCHITECTURE 129 the number of actions triggered in a specific time window (δ) in relation to an activation threshold (TAction ) implemented as a spontaneous activity (Equation A.2). Positive curiosity levels, caused by low action activity, lead to the inhibition of perception by a combination of spontaneous activity and an inhibitory reflex from the action set. The action network also has spontaneous activity, so that when all input is extinguished the camera will make some random movement. When perception is active it overrides the spontaneous activity (Equation A.3)). This intrinsic action by itself constitutes a kind of exploratory searching behavior set when no perception is available to lead the action. Random search is also regulated by the curiosity drive since the intrinsic actions will lower the curiosity level, which will allow perception to rule again. This system allows the gaze control to continuously track for faces and avoids getting stuck in missclassified points. In pseudo code the visualmotor gaze system could be described as: Perception Perception = 0 Drive = TAction − now−δ X Drive ≤ 0, (A.1) Drive > 0 Action (A.2) t=now Perception control Action = Random control Perception > 0, Perception = 0 (A.3) 130 APPENDIX A. INTERNAL DRIVE REGULATION OF SENSORIMOTOR REFLEXES Figure A.4: Navigation loop organization. Action has spontaneous activity. Action can be inhibited by the perception and by the drive. The drive is activated by the action. The navigation loop follows a different architecture when compared to the visual loop (Figure A.4). It receives sensory input from an array of proximity sensors positioned around the robot body and acts through motor commands that spin the wheels. The expected behavior is that the robot runs around the location, avoiding collisions of any kind. Differently to the vision loop, in the navigation loop the reflex from the perception to the action in inhibitory. Whenever an obstacle is perceived the motor control is inhibited. Motor action has spontaneous activity, which when is not inhibit-ed causes the robot to follow a random direction (Equation A.4). Direction is changed randomly and periodically with a time constant (σ) defined as 10 seconds. The drive in the navigation task works as a negative feedback. It is set in a way that the robot periodically stops at a certain position so that the guests have time to collect the load. The combination between the time constant (δ) and the drive activation threshold (TAction ) sets the time the robot takes in the stop phase and the time it stays in the moving phase (Equation A.5). A.3. RESULTS 131 0 Action = Random control Drive = now−δ X Perception or Drive > 0, (A.4) Otherwise Action − TAction (A.5) t=now A.3 Results A full demonstration of the living machine was performed during the Future and Emergent Technologies (FET’11) at the Polytechnic University of Warsaw in November of 2011 (Figure A.5). The robot was used to distribute chocolates to the attendees of the event during the coffee breaks. The venue consisted of a large hall of approximately 4000 square meter filled with tables and other sorts of obstacles. Before the demonstration, in order to profile the behavioral capability of the navigation loop and validate the parameters we used the control mechanism to steer a simulated robot in a virtual environment. For this experiment we used a drive time constant of 20 seconds and a TAction of on average 2.5 kilometers/hour or 5 degrees per second. The aim of the simulations was to verify the ability of the robot to cover the whole catering space and quantify the quality of its delivery service. We run five trials of three hours with different areas from 1600 m2 to 7000 m2 (Figure A.6). A specific area was considered “visited” when it was closer than 3.75 meters from a position where the robot was stationary (drive above zero). The control paradigm was able to deliver to more than 90% of the locations in all the arena sizes. Moreover, after 1 hour it was able 132 APPENDIX A. INTERNAL DRIVE REGULATION OF SENSORIMOTOR REFLEXES Figure A.5: Demonstration venue (right) and the robot (top-left), chocolates and candies are placed in a plate located in the back part of the robot body. to visit more than 60% of the arena with an area similar to the experimental site. The gaze system was also verified in lab conditions. TAction was set to .1 movements/second in average and the time constant to 10 seconds. This would imply that if no movement was done in 10 seconds, the drive would be active and gaze would follow a random direction. With controlled conditions - single face with white background – the vision loop was able to fix the gaze correctly in all the trials (n=10) and with the subject stationary it took an average time of 10.5 seconds to gaze in an-other direction. With the subject moving slowly (< 1 steps per second or 0.76 m/sec), the gaze system never lost track of the subject (n=10). In the case in which the subject was moving fast (> 1 steps per second), the gaze system always lost track of the user when the movement followed continuously in the same direction for more than 2 seconds (n=10). Movements confined to the view range were successfully tracked by the vision loop. Thus, the system is highly reliable in localizing stationary or slowly moving human faces with a speed of up to 3 km/h. A.3. RESULTS 133 Figure A.6: Simulated navigation data. (A) Time evolution (exponential fit) for the percentage of the area covered by delivery stops for five different arena sizes. (B) Sample of robot trajectories and delivery spot spatial distributions at different time windows for the 5921 m2 square arena. In the real demonstration, the robot was successful in the task of distributing chocolates. In total, 213 pieces were distributed in a time course of 10 hours or on the average about 1 piece every 3 minutes. The gaze system worked for the whole time course of the demonstration and was effective in calling attention and in creating a connection with the guests (Figure A.7). The robot was able to cover autonomously a large part of the hall’s sur- 134 APPENDIX A. INTERNAL DRIVE REGULATION OF SENSORIMOTOR REFLEXES Figure A.7: Demonstration of the gazing behavior by the visuomotor loop in a sequence with a moving subject. face, confirming the simulation results. The variability in the movements caused by the drive regulation showed to be effective since most of the guest would only approach the robot when it stopped or was moving slowly. We could observe situations in which it got stuck (N=19 in 10 hours). This would happen mostly in corners and in places with high desk and tables density. In these cases the navigation system was overridden by a remote control. Moreover, due to the different kind of materials used in the room, the proximity sensor could not detect a few obstacles causing the robot to crash. In moments with high guest density the navigation system was overridden for security reasons. Although both control loops were set independently and could not exchange any kind of messages through the computer, the influence of navigation loop over the gaze loop is striking. When the robot was in movement, gaze would keep directed to the same face for a longer time if compared to when the robot is still. The reason is that the movement of the robot forced the gaze system to activate the saccade movements more frequently, extending the time of drive integration. With the parameters used in the demonstration, the robot would keep the camera focused for about 10 seconds if the target face remained still. With the robot in A.4. DISCUSSION 135 movement (> 0.5 meters per second) the robot would only change the target face in case it lost the target face for a period greater than 1 second (caused mainly by visual obstruction). A.4 Discussion We addressed the question whether designing a controller that achieves its coherence through dynamic interaction with the environment can render robust and effective behavior. In this example of designing for emergence we targeted a delivery service task where a mobile robot had to deliver chocolate to a naive audience in a public event. The public demonstration of the presented control system is an example of how enhanced drive based reactive control can lead to emergent behavioral skills sufficient for permitting effective human-robots interaction. More specifically to the waiter task, it generates gaze and interpersonal distance regulation behaviors. The approach used is grounded on the regulation of reactive sensorimotor loops through internal drives that adds complexity to the performed activity and modulates the reactive response on an uncorrelated time-scale. Most importantly, the robot accomplishes its mission without the need of any declarative representation of the task or the other agents involved. Nevertheless, the demonstration showed that the system is sufficient but not optimal to reproduce a waiter performance. This is somehow expected since the sys-tem can be enhanced in many ways. But it is important to highlight the sufficiency of the reactive control since it gives a safe behaving procedure for any system eventually built on top of it. This form of environmentally mediated allostatic control can be seen as an hypothesis on how biological systems ultimately support survival in potentially harmful situations or to satisfy a range of needs (SanchezFibla et al., 2010). Moreover, it would be interesting to observe how 136 APPENDIX A. INTERNAL DRIVE REGULATION OF SENSORIMOTOR REFLEXES multiple robots would interact in the performance of the task. Since the covering of space tended to be confined to a certain region for a limited time, a simple communication channel in which robots avoid other machines might help to establish zones of action, reducing the active area of each robot and diminishing the time needed to cover the space through emergent collaboration. Regarding the extension of the control system, it is possible to keep the reactive idea by establishing direct communication between the two control loops. One example would be to use the face detection component to modulate the drive of the navigation, enhancing the sensation of approaching a guest. However, the clearest possibility is the addition of more cognitive skills such as memory and planning. The reactive system can support the construction of highly structured representations when considering the other two layers of the DAC architecture (Duff et al., 2010; Duff & Verschure, 2010). The same platform has been controlled by a full DAC architecture in non-interactive tasks in which we included features such as mapping, sequence learning, object recognition and spatial memory (Rennó-Costa et al., 2011). More specifically to the control loops, the visual-motor loop can be enhanced by face recognition itself (Luvizotto et al., 2011). This would allow the robot to focus on guests who have not yet been served. Regarding the navigation loop, the use of a spatial memory system would allow a homogeneous covering of the space since it would be possible to remember where it was before and avoid recently visited locations. Another possibility is to integrate both loops in the mapping of the space (Verschure et al., 2006). Bibliography Aggleton, J. P., Albasser, M. M., Aggleton, D. J., Poirier, G. L., & Pearce, J. M. (2010). Lesions of the rat perirhinal cortex spare the acquisition of a complex configural visual discrimination yet impair object recognition. Behavioral neuroscience, 124(1):55–68. ISSN 19390084. doi: 10.1037/a0018320. Amit, D. J. (1992). Modeling Brain Function: The World of Attractor Neural Networks. Cambridge University Press. ISBN 0521421241. Arleo, A. & Gerstner, W. (2000). Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. ical Cybernetics, 83(3):287–299. ISSN 0340-1200. Biolog- doi: 10.1007/ s004220000171. Arleo, A., Smeraldi, F., Hug, S., & Gerstner, W. (2001). Place Cells and Spatial Navigation based on Vision, Path Integration, and Reinforcement Learning. Dins T. K. Leen, T. G. Dietterich, & V. Tresp, editors, Advances in Neural Information Processing Systems 13, ps. 89–95. Path Integration, and Reinforcement Learning.&quot; Advances in Neural Information Processing Systems. Asama, H., Fukuda, T., & Arai, T. (1994). Distributed Autonomous Robotic Systems. Springer-Verlag. ISBN 0387701478. Astur, R. S., Taylor, L. B., Mamelak, A. N., Philpott, L., & Sutherland, R. J. (2002). Humans with hippocampus damage display severe spatial 138 BIBLIOGRAPHY 139 memory impairments in a virtual Morris water task. Behavioural Brain Research, 132(1):77–84. ISSN 01664328. doi: 10.1016/S0166-4328(01) 00399-0. Balkenius, C. & Morén, J. (2000). A Computational Model of Context Processing. Dins J.-A. Meyer, A. Berthoz, D. Floreano, H. L. Roitblat, & S. W. Wilson, editors, From Animals to Animats 6: Proceedings of the 6th International Conference on the Simulation of Adaptive Behaviour. The MIT Press, Cambridge, MA, USA. Bannerman, D. M., Yee, B. K., Good, M. A., Heupel, M. J., Iversen, S. D., & Rawlins, J. N. (1999). Double dissociation of function within the hippocampus: a comparison of dorsal, ventral, and complete hippocampal cytotoxic lesions. Behavioral neuroscience, 113(6):1170–88. ISSN 0735-7044. Barry, C. & Burgess, N. (2007). Learning in a geometric model of place cell firing. Hippocampus, 17(9):786–800. ISSN 1050-9631. doi: 10. 1002/hipo.20324. Barry, C., Hayman, R., Burgess, N., & Jeffery, K. J. (2007). Experience-dependent rescaling of entorhinal grids. Nature neuro- science, 10(6):682–4. ISSN 1097-6256. doi: 10.1038/nn1905. Barry, C., Lever, C., Hayman, R., Hartley, T., Burton, S., O’Keefe, J., Jeffery, K., & Burgess, N. (2006). The boundary vector cell model of place cell firing and spatial memory. Reviews in the neurosciences, 17(1-2):71–97. ISSN 0334-1763. Bartos, M., Vida, I., & Jonas, P. (2007). Synaptic mechanisms of synchronized gamma oscillations in inhibitory interneuron networks. Nature reviews. Neuroscience, 8(1):45–56. ISSN 1471-003X. doi: 10.1038/nrn2044. Bartsch, T., Schönfeld, R., Müller, F. J., Alfke, K., Leplow, B., Aldenhoff, J., Deuschl, G., & Koch, J. M. (2010). Focal lesions of human 140 BIBLIOGRAPHY hippocampal CA1 neurons in transient global amnesia impair place memory. Science (New York, N.Y.), 328(5984):1412–5. ISSN 10959203. doi: 10.1126/science.1188160. Bekey, G. A. (2005). Autonomous Robots: From Biological Inspiration to Implementation and Control. MIT Press. ISBN 9780262025782. Bernardet, U., i Badia, S. B., & Verschure, P. F. M. J. (2007). The eXperience Induction Machine and its Role in the Research on Presence. Dins The 10th Annual International Workshop on Presence. Bird, C. M. & Burgess, N. (2008). The hippocampus and memory: insights from spatial processing. Nature reviews. Neuroscience, 9(3):182– 94. ISSN 1471-0048. doi: 10.1038/nrn2335. Blair, H. T., Welday, A. C., & Zhang, K. (2007). Scale-invariant memory representations emerge from moiré interference between grid fields that produce theta oscillations: a computational model. The Journal of Neuroscience, 27(12):3211–29. ISSN 1529-2401. doi: 10.1523/JNEUROSCI.4724-06.2007. Blazquez, P. M., Hirata, Y., Heiney, S. A., Green, A. M., & Highstein, S. M. (2003). Cerebellar signatures of vestibulo-ocular reflex motor learning. The Journal of neuroscience : the official journal of the Society for Neuroscience, 23(30):9742–51. ISSN 1529-2401. Bliss, T. V. & Lø mo, T. (1973). Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path. The Journal of physiology, 232(2):331–56. ISSN 0022-3751. Bohbot, V. D., Kalina, M., Stepankova, K., Spackova, N., Petrides, M., & Nadel, L. (1998). Spatial memory deficits in patients with lesions to the right hippocampus and to the right parahippocampal cortex. Neuropsychologia, 36(11):1217–38. ISSN 0028-3932. BIBLIOGRAPHY 141 Bragin, A., Jandó, G., Nádasdy, Z., Hetke, J., Wise, K., & Buzsáki, G. (1995). Gamma (40-100 Hz) oscillation in the hippocampus of the behaving rat. The Journal of neuroscience : the official journal of the Society for Neuroscience, 15(1 Pt 1):47–60. ISSN 0270-6474. Braitenberg, V. (1984). Vehicles: Experiments in synthetic psychology. MIT Press, Cambridge, MA. Brotons-Mas, J. R., Montejo, N., O’Mara, S. M., & Sanchez-Vives, M. V. (2010). Stability of subicular place fields across multiple light and dark transitions. The European journal of neuroscience, 32(4):648–58. ISSN 1460-9568. doi: 10.1111/j.1460-9568.2010.07308.x. Brun, V. H., Leutgeb, S., Wu, H.-Q., Schwarcz, R., Witter, M. P., Moser, E. I., & Moser, M.-B. (2008a). Impaired Spatial Representation in CA1 after Lesion of Direct Input from Entorhinal Cortex. Neuron, 3(2):290– 302. ISSN 0896-6273. doi: 10.1016/j.neuron.2007.11.034. Brun, V. H., Solstad, T., Kjelstrup, K. B., Fyhn, M., Witter, M. P., Moser, E. I., & Moser, M.-B. (2008b). Progressive increase in grid scale from dorsal to ventral medial entorhinal cortex. Hippocampus, 18(12):1200–12. ISSN 1098-1063. doi: 10.1002/hipo.20504. Bunsey, M. & Eichenbaum, H. (1996). Conservation of hippocampal memory function in rats and humans. Nature, 379(6562):255–7. ISSN 0028-0836. doi: 10.1038/379255a0. Burgess, N., Barry, C., & O’Keefe, J. (2007). An oscillatory interference model of grid cell firing. Hippocampus, 17(9):801–12. ISSN 1050-9631. doi: 10.1002/hipo.20327. Burgess, N., Donnett, J. G., Jeffery, K. J., & O’Keefe, J. (1997). Robotic and neuronal simulation of the hippocampus and rat navigation. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 352(1360):1535–43. ISSN 0962-8436. doi: 10.1098/rstb.1997.0140. 142 BIBLIOGRAPHY Burwell, R. D. (2000). The parahippocampal region: corticocortical connectivity. Annals of the New York Academy of Sciences, 911:25–42. ISSN 0077-8923. Burwell, R. D. & Amaral, D. G. (1998). Perirhinal and postrhinal cortices of the rat: interconnectivity and connections with the entorhinal cortex. The Journal of comparative neurology, 391(3):293–321. ISSN 0021-9967. Burwell, R. D., Saddoris, M. P., Bucci, D. J., & Wiig, K. A. (2004). Corticohippocampal contributions to spatial and contextual learning. The Journal of neuroscience : the official journal of the Society for Neuroscience, 24(15):3826–36. ISSN 1529-2401. doi: 10.1523/JNEUROSCI. 0410-04.2004. Calhoun, J. B. (1963). The ecology and sociology of the Norway rat. U.S. Dept. of Health, Education, and Welfare, Public Health Service. Canto, C. B., Wouterlood, F. G., & Witter, M. P. (2008). What does the anatomical organization of the entorhinal cortex tell us? Neural plasticity, 2008:381243. ISSN 1687-5443. doi: 10.1155/2008/381243. Carlsen, J., De Olmos, J., & Heimer, L. (1982). Tracing of two-neuron pathways in the olfactory system by the aid of transneuronal degeneration: projections to the amygdaloid body and hippocampal formation. The Journal of comparative neurology, 208(2):196–208. ISSN 00219967. doi: 10.1002/cne.902080208. Chen, H. & Zhu, C.-a. (2005). Modeling the dynamics of biomimetic underwater robot fish. Dins IEEE International Conference on Robotics and Biomimetics - ROBIO, ps. 478–483. IEEE. ISBN 0-7803-9315-5. doi: 10.1109/ROBIO.2005.246314. Cohen, N. J. & Corkin, S. (1981). The amnesic patient H. M.: Learning and retention of cognitive skill. Society for Neuroscience Abstracts, 7:517–518. BIBLIOGRAPHY 143 Cohen, N. J. & Squire, L. R. (1980). Preserved learning and retention of pattern-analyzing skill in amnesia: dissociation of knowing how and knowing that. Science (New York, N.Y.), 210(4466):207–10. ISSN 0036-8075. Corkin, S. (1968). Acquisition of motor skill after bilateral me- dial temporal-lobe excision. Neuropsychologia, 6(3):255–265. ISSN 00283932. doi: 10.1016/0028-3932(68)90024-9. Corkin, S. (2002). What’s new with the amnesic patient H.M.? Nature reviews. Neuroscience, 3(2):153–60. ISSN 1471-003X. doi: 10.1038/ nrn726. Crapse, T. B. & Sommer, M. A. (2008). Corollary discharge across the animal kingdom. Nature reviews. Neuroscience, 9(8):587–600. ISSN 1471-0048. doi: 10.1038/nrn2457. Davis, H. (1996). Underestimating the rat’s intelligence. Brain research. Cognitive brain research, 3(3-4):291–8. ISSN 0926-6410. de Almeida, L., Idiart, M., & Lisman, J. E. (2007). Memory retrieval time and memory capacity of the CA3 network: role of gamma frequency oscillations. Learning & Memory, 14(11):795–806. ISSN 1549-5485. doi: 10.1101/lm.730207. de Almeida, L., Idiart, M., & Lisman, J. E. (2009a). A second function of gamma frequency oscillations: an E%-max winner-take-all mechanism selects which cells fire. The Journal of Neuroscience, 29(23):7497–7503. ISSN 1529-2401. doi: 10.1523/JNEUROSCI.6044-08.2009. de Almeida, L., Idiart, M., & Lisman, J. E. (2009b). The input-output transformation of the hippocampal granule cells: from grid cells to place fields. The Journal of Neuroscience, 29(23):7504–7512. ISSN 1529-2401. doi: 10.1523/JNEUROSCI.6048-08.2009. 144 BIBLIOGRAPHY de Almeida, L., Idiart, M., & Lisman, J. E. (2010). The single place fields of CA3 cells: A two-stage transformation from grid cells. Hippocampus, 22(2):200–8. ISSN 1098-1063. doi: 10.1002/hipo.20882. De Valois, R. L. & De Valois, K. K. (1990). Spatial Vision. Oxford Univ. Press, New York. Deadwyler, S. A., Bunn, T., & Hampson, R. E. (1996). Hippocampal ensemble activity during spatial delayed-nonmatch-to-sample performance in rats. Journal of Neuroscience, 16(1):354–372. Degenetais, E. (2003). Synaptic Influence of Hippocampus on Pyramidal Cells of the Rat Prefrontal Cortex: An In Vivo Intracellular Recording Study. Cerebral Cortex, 13(7):782–792. ISSN 1460-2199. doi: 10.1093/ cercor/13.7.782. Dijkstra, E. W. (1959). A note on two problems in connexion with graphs. Numerische Mathematik, 1(1):269–271. ISSN 0029-599X. doi: 10.1007/ BF01386390. Doeller, C. F., Barry, C., & Burgess, N. (2010). Evidence for grid cells in a human memory network. Nature, 463(7281):657–61. ISSN 1476-4687. doi: 10.1038/nature08704. Duff, A., Rennó-Costa, C., Marcos, E., Luvizotto, A., Giovannucci, A., Sanchez-Fibla, M., Bernardet, U., & Verschure, P. (2010). From Motor Learning to Interaction Learning in Robots, volum 264 de Studies in Computational Intelligence. Springer Berlin Heidelberg, Berlin, Heidelberg. ISBN 978-3-642-05180-7. doi: 10.1007/978-3-642-05181-4. Duff, A., Sanchez Fibla, M., & Verschure, P. F. M. J. (2011). A biologically based model for the integration of sensory-motor contingencies in rules and plans: A prefrontal cortex based extension of the Distributed Adaptive Control architecture. Brain research bulletin, 85(5):289–304. ISSN 1873-2747. doi: 10.1016/j.brainresbull.2010.11.008. BIBLIOGRAPHY 145 Duff, A. & Verschure, P. F. M. J. (2010). Unifying perceptual and behavioral learning with a correlative subspace learning rule. Neurocomputing, 73(10-12):1818–1830. ISSN 09252312. doi: 10.1016/j.neucom. 2009.11.048. Dupret, D., Pleydell-Bouverie, B., & Csicsvari, J. (2010). Rate remapping: when the code goes beyond space. Neuron, 68(6):1015–6. ISSN 1097-4199. doi: 10.1016/j.neuron.2010.12.011. Eichenbaum, H. (2000). Hippocampus: Mapping or memory? rent Biology, 10(21):R785–R787. ISSN 09609822. Cur- doi: 10.1016/ S0960-9822(00)00763-6. Eichenbaum, H. (2001). The hippocampus and declarative memory: cognitive mechanisms and neural codes. Behavioural brain research, 127(12):199–207. ISSN 0166-4328. Ekstrom, A. D., Kahana, M. J., Caplan, J. B., Fields, T. A., Isham, E. A., Newman, E. L., & Fried, I. (2003). Cellular networks underlying human spatial navigation. Nature, 425(6954):184–8. ISSN 1476-4687. doi: 10.1038/nature01964. Eng, K., Babler, A., Bernardet, U., Blanchard, M., Costa, M., Delbruck, T., Douglas, R., Hepp, K., Klein, D., Manzolli, J., Mintz, M., Roth, F., Rutishauser, U., Wassermann, K., Whatley, A., Wittmann, A., Wyss, R., & Verschure, P. F. (2003). Ada - intelligent space: an artificial creature for the SwissExpo.02. Dins IEEE International Conference on Robotics and Automation (ICRA), volum 3, ps. 4154–4159. IEEE. ISBN 0-7803-7736-2. doi: 10.1109/ROBOT.2003.1242236. Ennaceur, A., Neave, N., & Aggleton, J. P. (1997). Spontaneous object recognition and object location memory in rats: the effects of lesions in the cingulate cortices, the medial prefrontal cortex, the cingulum bundle and the fornix. Experimental brain research. Experimentelle 146 BIBLIOGRAPHY Hirnforschung. Expérimentation cérébrale, 113(3):509–19. ISSN 00144819. Erdem, U. M. & Hasselmo, M. (2012). A goal-directed spatial navigation model using forward trajectory planning based on grid cells. The European journal of neuroscience. ISSN 1460-9568. doi: 10.1111/j. 1460-9568.2012.08015.x. Etienne, A. S., Maurer, R., Berlie, J., Reverdin, B., Rowe, T., Georgakopoulos, J., & Séguinot, V. (1998). Navigation through vector addition. Nature, 396(6707):161–4. ISSN 0028-0836. doi: 10.1038/24151. Etienne, A. S., Maurer, R., Boulens, V., Levy, A., & Rowe, T. (2004). Resetting the path integrator: a basic condition for route-based navigation. The Journal of experimental biology, 207(Pt 9):1491–508. ISSN 0022-0949. Fenton, A. A., Csizmadia, G., & Muller, R. U. (2000a). Conjoint control of hippocampal place cell firing by two visual stimuli. I. The effects of moving the stimuli on firing field positions. The Journal of general physiology, 116(2):191–209. ISSN 0022-1295. Fenton, A. A., Csizmadia, G., & Muller, R. U. (2000b). Conjoint control of hippocampal place cell firing by two visual stimuli. II. A vectorfield theory that predicts modifications of the representation of the environment. The Journal of general physiology, 116(2):211–21. ISSN 0022-1295. Fenton, A. A., Lytton, W. W., Barry, J. M., Lenck-Santini, P.-P., Zinyuk, L. E., Kubík, S., Bures, J., Poucet, B., Muller, R. U., & Olypher, A. V. (2010). Attention-like modulation of hippocampus place cell discharge. The Journal of neuroscience : the official journal of the Society for Neuroscience, 30(13):4613–25. ISSN 1529-2401. doi: 10. 1523/JNEUROSCI.5576-09.2010. BIBLIOGRAPHY 147 Foster, D. J., Morris, R. G., & Dayan, P. (2000). A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus, 10(1):1–16. ISSN 1050-9631. doi: 10.1002/ (SICI)1098-1063(2000)10:1<1::AID-HIPO1>3.0.CO;2-1. Foster, D. J. & Wilson, M. A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440(7084):680–3. ISSN 1476-4687. doi: 10.1038/nature04587. Frank, L. M., Brown, E. N., & Wilson, M. (2000). Trajectory Encoding in the Hippocampus and Entorhinal Cortex. Neuron, 27(1):169–178. ISSN 08966273. doi: 10.1016/S0896-6273(00)00018-0. Fried, I., MacDonald, K. A., & Wilson, C. L. (1997). Single Neuron Activity in Human Hippocampus and Amygdala during Recognition of Faces and Objects. Neuron, 18(5):753–765. ISSN 08966273. doi: 10.1016/S0896-6273(00)80315-3. Fyhn, M., Hafting, T., Treves, A., Moser, M.-b., & Moser, E. I. (2007). Hippocampal remapping and grid realignment in entorhinal cortex. Nature, 446(7489):190–194. doi: 10.1038/nature05601. Fyhn, M., Molden, S., Witter, M. P., Moser, E. I., & Moser, M.B. (2004). Spatial representation in the entorhinal cortex. Sci- ence (New York, N.Y.), 305(5688):1258–64. ISSN 1095-9203. doi: 10.1126/science.1099901. Gandhi, N. J. & Katnani, H. A. (2011). Motor functions of the superior colliculus. Annual review of neuroscience, 34:205–31. ISSN 1545-4126. doi: 10.1146/annurev-neuro-061010-113728. Gaskin, S., Tremblay, A., & Mumby, D. G. (2003). Retrograde and anterograde object recognition in rats with hippocampal lesions. Hippocampus, 13(8):962–9. ISSN 1050-9631. doi: 10.1002/hipo.10154. 148 BIBLIOGRAPHY Gaussier, P., Revel, A., Banquet, J. P., & Babeau, V. (2002). From view cells and place cells to cognitive map learning: processing stages of the hippocampal system. Biological cybernetics, 86(1):15–28. ISSN 0340-1200. Giocomo, L. M., Moser, M.-B., & Moser, E. I. (2011). Computational models of grid cells. Neuron, 71(4):589–603. ISSN 1097-4199. doi: 10.1016/j.neuron.2011.07.023. Giraldeau, L.-A. & Caraco, T. (2000). Social Foraging Theory. Princeton University Press. ISBN 0691048762. Girardeau, G. & Zugaro, M. (2011). Hippocampal ripples and memory consolidation. Current opinion in neurobiology, 21(3):452–9. ISSN 1873-6882. doi: 10.1016/j.conb.2011.02.005. Gothard, K. M., Skaggs, W. E., & McNaughton, B. L. (1996). Dynamics of mismatch correction in the hippocampal ensemble code for space: interaction between path integration and environmental cues. Journal of Neuroscience, 16(24):8027–8040. Gray, J. A. & McNaughton, N. (2000). The Neuropsychology of Anxiety: An Enquiry into the Functions of the Septo-Hippocampal System (Oxford Science Publications). Oxford University Press, USA. ISBN 0198522703. Green, J. D. & Arduini, A. A. (1954). Hippocampal activity in arousal. J Neurophysiol, 17(6):533–557. Guanella, A. & Verschure, P. (2006). A Model of Grid Cells Based on a Path Integration Mechanism. Dins S. D. Kollias, A. Stafylopatis, W. Duch, & E. Oja, editors, Artificial Neural Networks – ICANN 2006, volum 4131 de Lecture Notes in Computer Science, ps. 740– 749. Springer Berlin Heidelberg, Berlin, Heidelberg, 4131 edició. ISBN 978-3-540-38625-4. doi: 10.1007/11840817. BIBLIOGRAPHY 149 Guanella, A. & Verschure, P. F. M. J. (2007). Prediction of the position of an animal based on populations of grid and place cells: a comparative simulation study. Journal of integrative neuroscience, 6(3):433–46. ISSN 0219-6352. Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005). Microstructure of a spatial map in the entorhinal cortex. Nature, 436:801–806. Hargreaves, E. L., Rao, G., Lee, I., & Knierim, J. J. (2005). Major Dissociation Between Medial and Lateral Entorhinal Input to Dorsal Hippocampus. Science, 1792(5729):1792–1794. doi: 10.1126/science. 1110449. Harrison, F. E., Reiserer, R. S., Tomarken, A. J., & McDonald, M. P. (2006). Spatial and nonspatial escape strategies in the Barnes maze. Learning & memory (Cold Spring Harbor, N.Y.), 13(6):809–19. ISSN 1072-0502. doi: 10.1101/lm.334306. Hartley, T., Burgess, N., O’Keefe, J., Lever, C., & Cacucci, F. (2000). Modeling Place Fields in Terms of the Cortical Inputs to the Hippocampus. Hippocampus, 10(4):369–79. ISSN 1050-9631. doi: 10. 1002/1098-1063(2000)10:4<369::AID-HIPO3>3.0.CO;2-0. Hasselmo, M. E. (2008). Grid cell mechanisms and function: contributions of entorhinal persistent spiking and phase resetting. Hippocampus, 18(12):1213–29. ISSN 1098-1063. doi: 10.1002/hipo.20512. Hasselmo, M. E. & Brandon, M. P. (2012). A model combining oscillations and attractor dynamics for generation of grid cell firing. Frontiers in Neural Circuits, 6(30). Hasselmo, M. E., Fransen, E., Dickson, C., & Alonso, A. A. (2000). Computational modeling of entorhinal cortex. Annals of the New York Academy of Sciences, 911:418–46. ISSN 0077-8923. 150 BIBLIOGRAPHY Hasselmo, M. E., Giocomo, L. M., & Zilli, E. A. (2007). Grid cell firing may arise from interference of theta frequency membrane potential oscillations in single neurons. Hippocampus, 17(12):1252–1271. ISSN 10509631. doi: 10.1002/hipo.20374. Hasselmo, M. E. & Stern, C. E. (2006). Mechanisms underlying working memory for novel information. Trends in cognitive sciences, 10(11):487–93. ISSN 1364-6613. doi: 10.1016/j.tics.2006.09.005. Hasslacher, B. & Tilden, M. W. (1995). Living machines. Robotics and Autonomous Systems, 15(1-2):143–169. ISSN 09218890. doi: 10.1016/ 0921-8890(95)00019-C. Hayman, R. M. & Jeffery, K. J. (2008). How heterogeneous place cell responding arises from homogeneous grids–a contextual gating hypothesis. Hippocampus, 18(12):1301–13. ISSN 1098-1063. doi: 10.1002/hipo.20513. Hebb, D. O. (1932). Studies of the organization of behavior. I. Behavior of the rat in a field orientation. Journal of Comparative Psychology, 25:333–353. Herrnstein, R. J. (1970). On the law of effect. Journal of the experimental analysis of behavior, 13(2):243–66. ISSN 0022-5002. Hofstötter, C., Mintz, M., & Verschure, P. F. M. J. (2002). The cerebellum in action: a simulation and robotics study. The European journal of neuroscience, 16(7):1361–76. ISSN 0953-816X. Hopfield, J. J. (1982). Neural Networks and Physical Systems with Emergent Collective Computational Abilities. Proceedings of the National Academy of Sciences, 79(8):2554–2558. ISSN 0027-8424. doi: 10.1073/pnas.79.8.2554. Hori, E., Nishio, Y., Kazui, K., Umeno, K., Tabuchi, E., Sasaki, K., Endo, S., Ono, T., & Nishijo, H. (2005). Place-related neural responses BIBLIOGRAPHY 151 in the monkey hippocampal formation in a virtual space. Hippocampus, 15(8):991–6. ISSN 1050-9631. doi: 10.1002/hipo.20108. Hough, G. E. & Bingman, V. P. (2004). Spatial response properties of homing pigeon hippocampal neurons: correlations with goal locations, movement between goals, and environmental context in a radial-arm arena. Journal of comparative physiology. A, Neuroethology, sensory, neural, and behavioral physiology, 190(12):1047–62. ISSN 0340-7594. doi: 10.1007/s00359-004-0562-z. Hung, C. P., Kreiman, G., Poggio, T., & DiCarlo, J. J. (2005). Fast readout of object identity from macaque inferior temporal cortex. Science (New York, N.Y.), 310(5749):863–6. ISSN 1095-9203. doi: 10.1126/science.1117593. Hyman, J. M., Zilli, E. A., Paley, A. M., & Hasselmo, M. E. (2005). Medial prefrontal cortex cells show dynamic modulation with the hippocampal theta rhythm dependent on behavior. Hippocampus 15: 739–749. Inderbitzin, M., Wierenga, S., Väljamäe, A., Bernardet, U., & Verschure, P. F. M. J. (2009). Social cooperation and competition in the mixed reality space eXperience Induction Machine XIM. Virtual Reality, 13(3):153–158. ISSN 1359-4338. doi: 10.1007/s10055-009-0119-0. Jeffery, K. J. (2011). Place Cells, Grid Cells, Attractors, and Remapping. Neural Plasticity. doi: 10.1155/2011/182602. Jensen, O. & Lisman, J. E. (1996). Novel lists of 7 +/- 2 known items can be reliably stored in an oscillatory short-term memory network: interaction with long-term memory. Learning & memory (Cold Spring Harbor, N.Y.), 3(2-3):257–63. ISSN 1072-0502. Jin, Y. & Meng, Y. (2011). Morphogenetic Robotics: An Emerging New Field in Developmental Robotics. IEEE Transactions on Systems, 152 BIBLIOGRAPHY Man, and Cybernetics, Part C (Applications and Reviews), 41(2):145– 160. ISSN 1094-6977. doi: 10.1109/TSMCC.2010.2057424. Johnson, A. & Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. The Journal of neuroscience : the official journal of the Society for Neuroscience, 27(45):12176–89. ISSN 1529-2401. doi: 10.1523/JNEUROSCI.3761-07. 2007. Johnston, D. & Amaral, D. G. (1998). Hippocampus. Dins G. M. Shepherd, editor, The synaptic organization of the brain, ps. 417–458. Oxford UP, New York, 4 edició. Jung, M. W. & McNaughton, B. L. (1993). Spatial selectivity of unit activity in the hippocampal granular layer. Hippocampus, 3(2):165–82. ISSN 1050-9631. doi: 10.1002/hipo.450030209. Knapp, M. L. & Hall, J. A. (2009). Nonverbal Communication in Human Interaction. Cengage Learning. ISBN 0495568694. Koene, R. A. & Hasselmo, M. E. (2007). First-in-first-out item replacement in a model of short-term memory based on persistent spiking. Cerebral cortex (New York, N.Y. : 1991), 17(8):1766–81. ISSN 10473211. doi: 10.1093/cercor/bhl088. Kulvicius, T., Tamosiunaite, M., Ainge, J., Dudchenko, P., & Wörgötter, F. (2008). Odor supported place cell model and goal navigation in rodents. Journal of computational neuroscience, 25(3):481–500. ISSN 1573-6873. doi: 10.1007/s10827-008-0090-x. Langston, R. F., Ainge, J. A., Couey, J. J., Canto, C. B., Bjerknes, T. L., Witter, M. P., Moser, E. I., & Moser, M.-B. (2010). Development of the spatial representation system in the rat. Science (New York, N.Y.), 328(5985):1576–80. ISSN 1095-9203. doi: 10.1126/science.1188210. BIBLIOGRAPHY 153 Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L., & Pennartz, C. M. A. (2009). Hippocampus leads ventral striatum in replay of place-reward information. PLoS biology, 7(8):e1000173. ISSN 1545-7885. doi: 10.1371/journal.pbio.1000173. Lawson, B. (2001). The Language of Space. Routledge Chapman & Hall. ISBN 0750652462. Lee, A. K. & Wilson, M. A. (2002). Memory of sequential experience in the hippocampus during slow wave sleep. Neuron, 36(6):1183–94. ISSN 0896-6273. Leonard, J. & Durrant-Whyte, H. (1991). Simultaneous map building and localization for an autonomous mobile robot. Dins Proceedings IROS ’91:IEEE/RSJ International Workshop on Intelligent Robots and Systems ’91, ps. 1442–1447. IEEE. ISBN 0-7803-0067-X. doi: 10.1109/ IROS.1991.174711. Lepora, N. F., Pearson, M. J., Mitchinson, B., Evans, M., Fox, C., Pipe, A., Gurney, K., & Prescott, T. J. (2010). Naive Bayes novelty detection for a moving robot with whiskers. Dins IEEE International Conference on Robotics and Biomimetics, ps. 131–136. IEEE. Leutgeb, J. K., Leutgeb, S., Moser, M.-B., & Moser, E. I. (2007). Pattern separation in the dentate gyrus and CA3 of the hippocampus. Science, 315(5814):961–966. ISSN 1095-9203. doi: 10.1126/science.1135801. Leutgeb, S., Leutgeb, J. K., Barnes, C. A., Moser, E. I., McNaughton, B. L., & Moser, M.-B. (2005). Independent codes for spatial and episodic memory in hippocampal neuronal ensembles. Science, 309(5734):619–23. ISSN 1095-9203. doi: 10.1126/science.1114037. Lienhart, R., Kuranov, E., & Pisarevsky, V. (2003). Empirical Analysis of Detection Cascades of Boosted Classifiers for Rapid Object Detection. IN DAGM 25TH PATTERN RECOGNITION SYMPOSIUM, ps. 297 – 304. 154 BIBLIOGRAPHY Lisman, J. & Idiart, M. (1995). Storage of 7 +/- 2 short-term memories in oscillatory subcycles. Science, 267(5203):1512–1515. ISSN 0036-8075. doi: 10.1126/science.7878473. Lisman, J. E. (1999). Relating hippocampal circuitry to function: Recall of memory sequences by reciprocal dentate-CA3 interactions. Neuron, 22(2):233–242. ISSN 0896-6273. Lisman, J. E. (2007). pocampus: Role of the dual entorhinal inputs to hip- a hypothesis based on cue/action (non-self/self) cou- plets. Progress in Brain Research, 163:615–626. ISSN 00796123. doi: 10.1016/S0079-6123(07)63033-7. Lisman, J. E., Talamini, L. M., & Raffone, A. (2005). Recall of memory sequences by interaction of the dentate and CA3: a revised model of the phase precession. Neural networks : the official journal of the International Neural Network Society, 18(9):1191–201. ISSN 0893-6080. doi: 10.1016/j.neunet.2005.08.008. Lø mo, T. (1966). Frequency potentiation of excitatory synaptic activity in the dentate area of the hippocampal formation. Acta Physiologica Scandinavica, 68(128). Louie, K. & Wilson, M. A. (2001). Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep. Neuron, 29(1):145–56. ISSN 0896-6273. Lowe, D. (1999). Object recognition from local scale-invariant features. Dins Proceedings of the Seventh IEEE International Conference on Computer Vision, ps. 1150–1157 vol.2. IEEE. ISBN 0-7695-0164-8. doi: 10.1109/ICCV.1999.790410. Lund, H. H. (1997). Robot-Animal Interaction. Dins W. Gerstner, A. Germond, M. Hasler, & J.-D. Nicoud, editors, Artificial Neural Networks — ICANN’97, volum 1327 de Lecture Notes in Computer BIBLIOGRAPHY 155 Science, ps. 745–750. Springer-Verlag, Berlin/Heidelberg. ISBN 3-54063631-5. doi: 10.1007/BFb0020124. Luvizotto, A., Rennó-Costa, C., Pattacini, U., & Verschure, P. F. M. J. (2011). The encoding of complex visual stimuli by a canonical model of the primary visual cortex: Temporal population code for face recognition on the iCub robot. Dins 2011 IEEE International Conference on Robotics and Biomimetics, ps. 313–318. IEEE, Phuket, Thailand. ISBN 9781457721373. doi: 10.1109/ROBIO.2011.6181304. Luvizotto, A., Rennó-Costa, C., & Verschure, P. F. M. J. (2012). A wavelet-based neural model to optimize and read out a temporal population code. Frontiers in computational neuroscience, 6:21. ISSN 1662-5188. doi: 10.3389/fncom.2012.00021. MacDonald, C. J., Lepage, K. Q., Eden, U. T., & Eichenbaum, H. (2011). Hippocampal "time cells" bridge the gap in memory for discontiguous events. Neuron, 71(4):737–49. ISSN 1097-4199. doi: 10.1016/j.neuron. 2011.07.012. MacDonall, J. S., Goodell, J., & Juliano, A. (2006). Momentary maximizing and optimal foraging theories of performance on concurrent VR schedules. Behavioural Processes, 72(3):283–299. Manns, J. R. & Eichenbaum, H. (2006). Evolution of Declarative Memory. Hippocampus, 16(9):795–808. Manns, J. R. & Squire, L. R. (2001). Perceptual learning, awareness, and the hippocampus. Hippocampus, 11(6):776–82. ISSN 1050-9631. doi: 10.1002/hipo.1093. Manzolli, J. & Verschure, P. F. M. J. (2005). Roboser: A Real-World Composition System. Computer Music Journal, 29(3):55–74. ISSN 0148-9267. doi: 10.1162/0148926054798133. 156 BIBLIOGRAPHY Marcos, E., Duff, A., Sanchez-Fibla, M., & Verschure, P. F. M. J. (2010). The neuronal substrate underlying order and interval representations in sequential tasks: A biologically based robot study. Dins IEEE/IJCNN International Joint Conference on Neural Networks, ps. 1–8. IEEE. ISBN 978-1-4244-6916-1. doi: 10.1109/IJCNN.2010.5596919. Markus, E. J., Barnes, C. A., McNaughton, B. L., Gladden, V. L., & Skaggs, W. E. (1994). Spatial information content and reliability of hippocampal CA1 neurons: effects of visual input. Hippocampus, 4(4):410–21. ISSN 1050-9631. doi: 10.1002/hipo.450040404. Marr, D. (1971). Simple Memory: A Theory for Archicortex. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences (1934-1990), 262(841):23–81. ISSN 0080-4622. doi: 10.1098/rstb.1971.0078. Mathews, Z., Lechon, M., Calvo, J. B., Dhir, A., Duff, A., Bermudez i Badia, S., & Verschure, P. F. (2009). Insect-Like mapless navigation based on head direction cells and contextual learning using chemovisual sensors. Dins IEEE/RSJ International Conference on Intelligent Robots and Systems, ps. 2243–2250. IEEE. ISBN 978-1-4244-3803-7. doi: 10.1109/IROS.2009.5354264. Mazer, J. A., Vinje, W. E., McDermott, J., Schiller, P. H., & Gallant, J. L. (2002). Spatial frequency and orientation tuning dynamics in area V1. Proceedings of the National Academy of Sciences of the United States of America, 99(3):1645–50. ISSN 0027-8424. doi: 10.1073/pnas. 022638499. McDonald, A. J. & Mascagni, F. (1996). Cortico-cortical and corticoamygdaloid projections of the rat occipital cortex: a Phaseolus vulgaris leucoagglutinin study. Neuroscience, 71(1):37–54. ISSN 0306-4522. McDonald, R. J., Jones, J., Richards, B., & Hong, N. S. (2006). A double dissociation of dorsal and ventral hippocampal function on a BIBLIOGRAPHY 157 learning and memory task mediated by the dorso-lateral striatum. The European journal of neuroscience, 24(6):1789–801. ISSN 0953-816X. doi: 10.1111/j.1460-9568.2006.05064.x. McHaffie, J. G., Stanford, T. R., Stein, B. E., Coizet, V., & Redgrave, P. (2005). Subcortical loops through the basal ganglia. Trends in neurosciences, 28(8):401–7. ISSN 0166-2236. doi: 10.1016/j.tins.2005. 06.006. McNaughton, B. L., Battaglia, F. P., Jensen, O., Moser, E. I., & Moser, M.-B. (2006). Path integration and the neural basis of the ’cognitive map’. Nature reviews. Neuroscience, 7(8):663–78. ISSN 1471-003X. doi: 10.1038/nrn1932. McNaughton, B. L., O’Keefe, J., & Barnes, C. A. (1983). The stereotrode: a new technique for simultaneous isolation of several single units in the central nervous system from multiple unit records. Journal of neuroscience methods, 8(4):391–7. ISSN 0165-0270. Michaud, F., Brosseau, Y., Cote, C., Letourneau, D., Moisan, P., Ponchon, A., Raievsky, C., Valin, J.-M., Beaudryy, E., & Kabanza, F. (2005). Modularity and integration in the design of a socially interactive robot. Dins IEEE International Workshop on Robot and Human Interactive Communication., ps. 172–177. IEEE. ISBN 0-7803-9274-4. doi: 10.1109/ROMAN.2005.1513775. Min, M.-Y., Asztely, F., Kokaia, M., & Kullmann, D. M. (1998). Longterm potentiation and dual-component quantal signaling in the dentate gyrus. Proceedings of the National Academy of Sciences of the United States of America, 95(April):4702–4707. Morris, R. G. M., Garrud, P., Rawlins, J. N. P., & O’Keefe, J. (1982). Place navigation impaired in rats with hippocampal lesions. Nature, 297(5868):681–683. ISSN 0028-0836. doi: 10.1038/297681a0. 158 BIBLIOGRAPHY Muller, R. (1996). A quarter of a century of place cells. 17(5):813–22. ISSN 0896-6273. Neuron, Muller, R., Bostock, E., Taube, J., & Kubie, J. (1994). On the directional firing properties of hippocampal place cells. J. Neurosci., 14(12):7235– 7251. Muller, R. & Kubie, J. (1987). The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells. J. Neurosci., 7(7):1951–1968. Muller, R. U., Kubie, J. L., Bostock, E. M., Taube, J. S., & Quirk, G. J. (1991). Spatial firing correlates of neurons in the hippocampal formation of freely moving rats. Dins J. Paillard, editor, Brain and Space, ps. 296–333. Oxford University Press. ISBN 0198542844. Mumby, D. G., Gaskin, S., Glenn, M. J., Schramek, T. E., & Lehmann, H. (2002). Hippocampal damage and exploratory preferences in rats: memory for objects, places, and contexts. Learning & memory (Cold Spring Harbor, N.Y.), 9(2):49–57. ISSN 1072-0502. doi: 10.1101/lm. 41302. Murray, E. A., Bussey, T. J., & Saksida, L. M. (2007). Visual perception and memory: a new view of medial temporal lobe function in primates and rodents. Annual review of neuroscience, 30:99–122. ISSN 0147006X. doi: 10.1146/annurev.neuro.29.051605.113046. Myhrer, T. (1988). Exploratory behavior and reaction to novelty in rats with hippocampal perforant path systems disrupted. Behavioral neuroscience, 102(3):356–62. ISSN 0735-7044. Nadel, L. (1991). The hippocampus and space revisited. Hippocampus, 1(3):221–9. ISSN 1050-9631. doi: 10.1002/hipo.450010302. BIBLIOGRAPHY 159 Nadel, L. & Moscovitch, M. (1997). Memory consolidation, retrograde amnesia and the hippocampal complex. Current opinion in neurobiology, 7(2):217–27. ISSN 0959-4388. Nafstad, P. H. J. (1967). An electron microscope study on the termination of the perforant path fibres in the hippocampus and the fascia dentata. Zeitschrift für Zellforschung und Mikroskopische Anatomie, 76(4):532– 542. ISSN 0302-766X. doi: 10.1007/BF00339754. Navratilova, Z., Giocomo, L. M., Fellous, J.-M., Hasselmo, M. E., & McNaughton, B. L. (2011). Phase precession and variable spatial scaling in a periodic attractor map model of medial entorhinal grid cells with realistic after-spike dynamics. Hippocampus, 22(4):772–89. ISSN 1098-1063. doi: 10.1002/hipo.20939. Navratilova, Z., Hoang, L. T., Schwindel, C. D., Tatsuno, M., & McNaughton, B. L. (2012). Experience-dependent firing rate remapping generates directional selectivity in hippocampal place cells. Frontiers in neural circuits, 6:6. ISSN 1662-5110. doi: 10.3389/fncir.2012.00006. O’Keefe, J., Burgess, N., Donnett, J. G., Jeffery, K. J., & Maguire, E. A. (1998). Place cells, navigational accuracy, and the human hippocampus. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 353(1373):1333–40. ISSN 0962-8436. doi: 10.1098/rstb.1998.0287. O’Keefe, J. & Conway, D. (1978). Hippocampal place units in the freely moving rat: Why they fire where they fire. Experimental Brain Research, 31(4):573–590. ISSN 0014-4819. doi: 10.1007/BF00239813. O’Keefe, J. & Dostrovsky, J. (1971). The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain research, 34(1):171–5. ISSN 0006-8993. O’Keefe, J. & Nadel, L. (1978). The Hippocampus as a Cognitive Map. Oxford University Press. 160 BIBLIOGRAPHY O’Keefe, J. & Recce, M. L. (1993). Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus, 3(3):317–30. ISSN 1050-9631. doi: 10.1002/hipo.450030307. Pennartz, C. M. A., Ito, R., Verschure, P. F. M. J., Battaglia, F. P., & Robbins, T. W. (2011). The hippocampal-striatal axis in learning, prediction and goal-directed behavior. Trends in neurosciences, 34(10):548–59. ISSN 1878-108X. doi: 10.1016/j.tins.2011.08.001. Pfeifer, R. & Bongard, J. (2006). How the Body Shapes the Way We Think: A New View of Intelligence (Bradford Books). A Bradford Book. ISBN 0262162393. Pfeifer, R. & Verschure, P. (1994). The challenge of autonomous agents: Pitfalls and how to avoid them. Autonomous Agents, ps. 237 – 263. Pitkänen, A., Pikkarainen, M., Nurminen, N., & Ylinen, A. (2000). Reciprocal connections between the amygdala and the hippocampal formation, perirhinal cortex, and postrhinal cortex in rat. A review. Annals of the New York Academy of Sciences, 911:369–91. ISSN 00778923. Pöschel, B., Draguhn, A., & Heinemann, U. (2002). Glutamate- induced gamma oscillations in the dentate gyrus of rat hippocampal slices. Brain Research, 938(1-2):22–28. ISSN 00068993. doi: 10.1016/S0006-8993(02)02477-0. Prim, R. C. (1957). Shortest connection networks and some generalizations. Bell System Technical Journal, 36:1389–1401. Quirk, G., Muller, R., & Kubie, J. (1990). The firing of hippocampal place cells in the dark depends on the rat’s recent experience. J. Neurosci., 10(6):2008–2017. BIBLIOGRAPHY 161 Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C., & Fried, I. (2005). Invariant visual representation by single neurons in the human brain. Nature, 435(7045):1102–7. ISSN 1476-4687. doi: 10.1038/nature03687. Ramón y Cajal, S. (1909). Histologie du système nerveux de l’homme & des vertébrés. Paris : Maloine. Recce, M. & O’Keefe, J. (1989). The tetrode: a new technique for multi-unit extracellular recording. Society for Neuroscience Abstracts, 15:1250. Rennó-Costa, C., Lisman, J. E., & Verschure, P. F. M. J. (2010a). The mechanism of rate remapping in the dentate gyrus. Neuron, 68(6):1051–8. ISSN 1097-4199. doi: 10.1016/j.neuron.2010.11.024. Rennó-Costa, C., Lisman, J. E., & Verschure, P. F. M. J. (2010b). The mechanism of rate remapping in the dentate gyrus. Dins Society for Neuroscience Abstracts. San Diego, CA, USA. Rennó-Costa, C., Lisman, J. E., & Verschure, P. F. M. J. (2012a). The mechanism of attractor dynamics in the CA3. In Preparation. Rennó-Costa, C., Lisman, J. E., & Verschure, P. F. M. J. (2012b). The mechanism of attractor dynamics in the CA3. Dins Society for Neuroscience Abstracts. New Orleans, LA, USA. Rennó-Costa, C., Luvizotto, A., Betella, A., Sanchez Fibla, M., & Verschure, P. F. M. J. (2012c). Internal drive regulation of sensorimotor reflexes in the control of a catering assistant autonomous robot. Dins Lecture Notes in Artificial Intelligence: Living Machines. Rennó-Costa, C., Luvizotto, A., Marcos, E., Duff, A., Sanchez-Fibla, M., & Verschure, P. F. M. J. (2011). Integrating neuroscience-based models towards an autonomous biomimetic Synthetic Forager. Dins 2011 IEEE International Conference on Robotics and Biomimetics, 162 BIBLIOGRAPHY ps. 210–215. IEEE, Phuket, Thailand. ISBN 978-1-4577-2138-0. doi: 10.1109/ROBIO.2011.6181287. Rennó-Costa, C. & Verschure, P. F. M. J. (2012). Nonspatial selectivity of place cells supports quasi-optimal behavior in mixed spatial/nonspatial tasks. In Preparation. Rivard, B., Li, Y., Lenck-Santini, P.-P., Poucet, B., & Muller, R. U. (2004). Representation of objects in space by two classes of hippocampal pyramidal cells. The Journal of general physiology, 124(1):9–25. ISSN 0022-1295. doi: 10.1085/jgp.200409015. Roberts, W. (1992). Foraging by rats on a radial maze:Learning, memory, and decision rules. Dins I. Gormezano & E. Wasserman, editors, Learning and memory: The behavioral and biological substrates, ps. 7–24. Erlbaum, Hillsdale, NJ. ISBN 0805808884. Rolls, E. T. (2007). An attractor network in the hippocampus: theory and neurophysiology. Learning & memory (Cold Spring Harbor, N.Y.), 14(11):714–31. ISSN 1549-5485. doi: 10.1101/lm.631207. Rolls, E. T., Stringer, S. M., & Elliot, T. (2006). Entorhinal cortex grid cells can map to hippocampal place cells by competitive learning. Network (Bristol, England), 17(4):447–65. ISSN 0954-898X. doi: 10. 1080/09548980601064846. Rolls, E. T., Xiang, J., & Franco, L. (2005). Object, space, and objectspace representations in the primate hippocampus. Journal of neurophysiology, 94(1):833–44. ISSN 0022-3077. doi: 10.1152/jn.01063.2004. Samsonovich, A. & McNaughton, B. L. (1997). Path Integration and Cognitive Mapping in a Continuous Attractor Neural Network Model. J. Neurosci., 17(15):5900–5920. Sanchez-Fibla, M., Bernardet, U., Wasserman, E., Pelc, T., Mintz, M., Jackson, J. C., Lansink, C., Pennartz, C., & Verschure, P. F. M. J. BIBLIOGRAPHY 163 (2010). Allostatic Control for Robot Behavior Regulation: a Comparative Rodent-Robot Study. Advances in Complex Systems, 13(3):377. ISSN 02195259. doi: 10.1142/S0219525910002621. Sargolini, F., Fyhn, M., Hafting, T., McNaughton, B. L., Witter, M. P., Moser, M.-B., & Moser, E. I. (2006). Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science (New York, N.Y.), 312(5774):758–62. ISSN 1095-9203. doi: 10.1126/science. 1125572. Savelli, F. & Knierim, J. J. (2010). Hebbian analysis of the transformation of medial entorhinal grid-cell inputs to hippocampal place fields. Journal of neurophysiology, 103(6):3167–3183. ISSN 1522-1598. doi: 10.1152/jn.00932.2009. Schenk, F., Inglin, F., & Gyger, M. (1983). Activity and exploratory behavior after lesions of the medial entorhinal cortex in the woodmouse (Apodemus sylvaticus). Behavioral and neural biology, 37(1):89–107. ISSN 0163-1047. Schweighofer, N., Arbib, M. A., & Dominey, P. F. (1996). A model of the cerebellum in adaptive control of saccadic gain. I. The model and its biological substrate. Biological cybernetics, 75(1):19–28. ISSN 0340-1200. Scoville, W. B. & Milner, B. (1957). Loss of recent memory after bilateral hippocampal lesions. Journal of neurology, neurosurgery, and psychiatry, 20(1):11–21. ISSN 0022-3050. Sharp, P. E. (2006). Subicular place cells generate the same "map" for different environments: comparison with hippocampal cells. Behavioural brain research, 174(2):206–14. ISSN 0166-4328. doi: 10.1016/j.bbr. 2006.05.034. 164 BIBLIOGRAPHY Si, B. & Treves, A. (2009). The role of competitive learning in the generation of DG fields from EC inputs. Cognitive neurodynamics, 3(2):177–87. ISSN 1871-4080. doi: 10.1007/s11571-009-9079-z. Simon, H. A. (1969). The architecture of complexity. Dins The Sciences of the Artificial, ps. 192–229. MIT Press, Cambridge, MA. Simons, D. J. & Levin, D. T. (1998). Failure to detect changes to people during a real-world interaction. Psychonomic Bulletin & Review, 5(4):644–649. ISSN 1069-9384. doi: 10.3758/BF03208840. Skaggs, W. E., McNaughton, B. L., Gothard, K. M., & Markus, E. J. (1993). An Information-Theoretic Approach to Deciphering the Hippocampal Code. Dins In Advances in Neural Information Processing Systems 5, [NIPS Conference], 1990, ps. 1030–1037. Morgan Kaufmann. doi: 10.1.1.52.5231. Skaggs, W. E., McNaughton, B. L., Wilson, M. A., & Barnes, C. A. (1996). Theta phase precession in hippocampal neuronal populations and the compression of temporal sequences. Hippocampus, 6(2):149– 172. ISSN 1050-9631. doi: 10.1002/(SICI)1098-1063(1996)6:2<149:: AID-HIPO6>3.0.CO;2-K. Solstad, T., Boccara, C. N., Kropff, E., Moser, M.-B., & Moser, E. I. (2008). Representation of geometric borders in the entorhinal cortex. Science (New York, N.Y.), 322(5909):1865–8. ISSN 1095-9203. doi: 10.1126/science.1166466. Solstad, T., Moser, E. I., & Einevoll, G. T. (2006). From grid cells to place cells: a mathematical model. Hippocampus, 16(12):1026–31. ISSN 1050-9631. doi: 10.1002/hipo.20244. Soltesz, I. & Deschênes, M. (1993). Low- and high-frequency membrane potential oscillations during theta activity in CA1 and CA3 pyramidal neurons of the rat hippocampus under ketamine-xylazine anesthesia. Journal of neurophysiology, 70(1):97–116. ISSN 0022-3077. BIBLIOGRAPHY 165 Song, J.-H., Rafal, R., & McPeek, R. (2010). Neural substrates of target selection for reaching movements in superior colliculus. Journal of Vision, 10(7):1082–1082. ISSN 1534-7362. doi: 10.1167/10.7.1082. Song, S. & Waldron, K. J. (1989). Machines that walk: The adaptive suspension vehicle. MIT Press, Cambridge, MA. Stephens, D. W., Brown, J. S., & Ydenberg, R. C. (2007). Foraging: Behavior and Ecology. University of Chicago Press, Chicago. Taube, J., Muller, R., & Ranck, J. (1990a). Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis. J. Neurosci., 10(2):420–435. Taube, J. S., Muller, R. U., & Ranck, J. B. (1990b). Head-direction cells recorded from the postsubiculum in freely moving rats. II. Effects of environmental manipulations. The Journal of neuroscience : the official journal of the Society for Neuroscience, 10(2):436–47. ISSN 0270-6474. Tolman, E. C. (1948). Cognitive Maps in Rats and Men. The Psychological Review, 55(4):189–208. Towers, S. K., LeBeau, F. E. N., Gloveli, T., Traub, R. D., Whittington, M. A., & Buhl, E. H. (2002). Fast Network Oscillations in the Rat Dentate Gyrus In Vitro. J Neurophysiol, 87(2):1165–1168. Trommald, M. & Hulleberg, G. (1997). Dimensions and density of dendritic spines from rat dentate granule cells based on reconstructions from serial electron micrographs. The Journal of comparative neurology, 377(1):15–28. ISSN 0021-9967. Tsoar, A., Nathan, R., Bartan, Y., Vyssotski, A., Dell’Omo, G., & Ulanovsky, N. (2011). Large-scale navigational map in a mammal. Proceedings of the National Academy of Sciences of the United States 166 BIBLIOGRAPHY of America, 108(37):E718–24. ISSN 1091-6490. doi: 10.1073/pnas. 1107365108. Ulanovsky, N. & Moss, C. F. (2007). Hippocampal cellular and network activity in freely moving echolocating bats. Nature neuroscience, 10(2):224–33. ISSN 1097-6256. doi: 10.1038/nn1829. Van Cauter, T., Camon, J., Alvernhe, A., Elduayen, C., Sargolini, F., & Save, E. (2012). Distinct Roles of Medial and Lateral Entorhinal Cortex in Spatial Cognition. Cerebral cortex (New York, N.Y. : 1991), ps. bhs033–. ISSN 1460-2199. doi: 10.1093/cercor/bhs033. Van Cauter, T., Poucet, B., & Save, E. (2008). Unstable CA1 place cell representation in rats with entorhinal cortex lesions. The European journal of neuroscience, 27(8):1933–46. ISSN 1460-9568. doi: 10.1111/ j.1460-9568.2008.06158.x. Vanderwolf, C. H. (1969). Hippocampal electrical activity and voluntary movement in the rat. Electroencephalography and clinical neurophysiology, 26(4):407–18. ISSN 0013-4694. Verschure, P. F. M. J. & Althaus, P. (2003). A real-world rational agent: unifying old and new AI. Cognitive Science, 27(4):30. doi: 10.1016/ S0364-0213(03)00034-X. Verschure, P. F. M. J., Kröse, B. J. A., & Pfeifer, R. (1992). Distributed Adaptive Control: The self-organization of structured behavior. Robotics and Autonomous Systems, 9(3):181–196. ISSN 09218890. doi: 10.1016/0921-8890(92)90054-3. Verschure, P. F. M. J., Voegtlin, T., & Douglas, R. J. (2003). Environmentally mediated synergy between perception and behaviour in mobile robots. Nature, 425:620–624. BIBLIOGRAPHY 167 Verschure, P. F. M. J., Wyss, R., & König, P. (2006). A Model of the Ventral Visual System Based on Temporal Stability and Local Memory. PLOS Biology, 4(5):1–8. doi: 10.1371/Citation. Walton, M. E., Bannerman, D. M., & Rushworth, M. F. S. (2002). The Role of Rat Medial Frontal Cortex in Effort-Based Decision Making. J. Neurosci., 22(24):10996–11003. Whitlock, J. R. & Derdikman, D. (2012). Head direction maps remain stable despite grid map fragmentation. Frontiers in Neural Circuits, 6. ISSN 1662-5110. doi: 10.3389/fncir.2012.00009. Wiener, S. I., Korshunov, V. A., Garcia, R., & Berthoz, A. (1995). Inertial, substratal and landmark cue control of hippocampal CA1 place cell activity. The European journal of neuroscience, 7(11):2206–19. ISSN 0953-816X. Witter, M. P. & Moser, E. I. (2006). Spatial representation and the architecture of the entorhinal cortex. Trends in neurosciences, 29(12):671–8. ISSN 0166-2236. doi: 10.1016/j.tins.2006.10.003. Witter, M. P., Naber, P. A., van Haeften, T., Machielsen, W. C., Rombouts, S. A., Barkhof, F., Scheltens, P., & Lopes da Silva, F. H. (2000a). Cortico-hippocampal communication by way of parallel parahippocampal-subicular pathways. Hippocampus, 10(4):398– 410. ISSN 1050-9631. doi: 10.1002/1098-1063(2000)10:4<398:: AID-HIPO6>3.0.CO;2-K. Witter, M. P., Van Hoesen, G. W., & Amaral, D. G. (1989). Topographical organization of the entorhinal projection to the dentate gyrus of the monkey. The Journal of neuroscience : the official journal of the Society for Neuroscience, 9(1):216–28. ISSN 0270-6474. Witter, M. P., Wouterlood, F. G., Naber, P. A., & Van Haeften, T. (2000b). Anatomical organization of the parahippocampal- 168 BIBLIOGRAPHY hippocampal network. Annals of the New York Academy of Sciences, 911:1–24. ISSN 0077-8923. Wood, E. R., Dudchenko, P. A., & Eichenbaum, H. (1999). The global record of memory in hippocampal neuronal activity. Nature, 397(6720):613–6. ISSN 0028-0836. doi: 10.1038/17605. Wood, E. R., Dudchenko, P. A., Robitsek, R. J., & Eichenbaum, H. (2000). Hippocampal Neurons Encode Information about Different Types of Memory Episodes Occurring in the Same Location. Neuron, 27(3):623–633. Wyss, R., Konig, P., & Verschure, P. F. M. J. (2003). Invariant representations of visual patterns in a temporal population code. Proceedings of the National Academy of Sciences of the United States of America, 100(1):324–9. ISSN 0027-8424. doi: 10.1073/pnas.0136977100. Zilli, E. A. (2012). Models of grid cell spatial firing published 2005–2011. Frontiers in Neural Circuits, 6(16).