Learning touch preferences with a tactile robot using dopamine modulated STDP in a model of insular cortex
Identifieur interne : 000187 ( Pmc/Curation ); précédent : 000186; suivant : 000188Learning touch preferences with a tactile robot using dopamine modulated STDP in a model of insular cortex
Auteurs : Ting-Shuo Chou [États-Unis] ; Liam D. Bucci [États-Unis] ; Jeffrey L. Krichmar [États-Unis]Source :
- Frontiers in Neurorobotics [ 1662-5218 ] ; 2015.
Abstract
Neurorobots enable researchers to study how behaviors are produced by neural mechanisms in an uncertain, noisy, real-world environment. To investigate how the somatosensory system processes noisy, real-world touch inputs, we introduce a neurorobot called CARL-SJR, which has a full-body tactile sensory area. The design of CARL-SJR is such that it encourages people to communicate with it through gentle touch. CARL-SJR provides feedback to users by displaying bright colors on its surface. In the present study, we show that CARL-SJR is capable of learning associations between conditioned stimuli (CS; a color pattern on its surface) and unconditioned stimuli (US; a preferred touch pattern) by applying a spiking neural network (SNN) with neurobiologically inspired plasticity. Specifically, we modeled the primary somatosensory cortex, prefrontal cortex, striatum, and the insular cortex, which is important for hedonic touch, to process noisy data generated directly from CARL-SJR's tactile sensory area. To facilitate learning, we applied dopamine-modulated Spike Timing Dependent Plasticity (STDP) to our simulated prefrontal cortex, striatum, and insular cortex. To cope with noisy, varying inputs, the SNN was tuned to produce traveling waves of activity that carried spatiotemporal information. Despite the noisy tactile sensors, spike trains, and variations in subject hand swipes, the learning was quite robust. Further, insular cortex activities in the incremental pathway of dopaminergic reward system allowed us to control CARL-SJR's preference for touch direction without heavily pre-processed inputs. The emerged behaviors we found in this model match animal's behaviors wherein they prefer touch in particular areas and directions. Thus, the results in this paper could serve as an explanation on the underlying neural mechanisms for developing tactile preferences and hedonic touch.
Url:
DOI: 10.3389/fnbot.2015.00006
PubMed: 26257639
PubMed Central: 4510776
Links toward previous steps (curation, corpus...)
- to stream Pmc, to step Corpus: Pour aller vers cette notice dans l'étape Curation :000187
Links to Exploration step
PMC:4510776Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Learning touch preferences with a tactile robot using dopamine modulated STDP in a model of insular cortex</title>
<author><name sortKey="Chou, Ting Shuo" sort="Chou, Ting Shuo" uniqKey="Chou T" first="Ting-Shuo" last="Chou">Ting-Shuo Chou</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><institution>Department of Computer Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea></wicri:regionArea>
<wicri:regionArea># see nlm:aff region in country</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Bucci, Liam D" sort="Bucci, Liam D" uniqKey="Bucci L" first="Liam D." last="Bucci">Liam D. Bucci</name>
<affiliation wicri:level="1"><nlm:aff id="aff2"><institution>Department of Cognitive Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea></wicri:regionArea>
<wicri:regionArea># see nlm:aff region in country</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Krichmar, Jeffrey L" sort="Krichmar, Jeffrey L" uniqKey="Krichmar J" first="Jeffrey L." last="Krichmar">Jeffrey L. Krichmar</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><institution>Department of Computer Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea></wicri:regionArea>
<wicri:regionArea># see nlm:aff region in country</wicri:regionArea>
</affiliation>
<affiliation wicri:level="1"><nlm:aff id="aff2"><institution>Department of Cognitive Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea></wicri:regionArea>
<wicri:regionArea># see nlm:aff region in country</wicri:regionArea>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PMC</idno>
<idno type="pmid">26257639</idno>
<idno type="pmc">4510776</idno>
<idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4510776</idno>
<idno type="RBID">PMC:4510776</idno>
<idno type="doi">10.3389/fnbot.2015.00006</idno>
<date when="2015">2015</date>
<idno type="wicri:Area/Pmc/Corpus">000187</idno>
<idno type="wicri:Area/Pmc/Curation">000187</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Learning touch preferences with a tactile robot using dopamine modulated STDP in a model of insular cortex</title>
<author><name sortKey="Chou, Ting Shuo" sort="Chou, Ting Shuo" uniqKey="Chou T" first="Ting-Shuo" last="Chou">Ting-Shuo Chou</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><institution>Department of Computer Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea></wicri:regionArea>
<wicri:regionArea># see nlm:aff region in country</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Bucci, Liam D" sort="Bucci, Liam D" uniqKey="Bucci L" first="Liam D." last="Bucci">Liam D. Bucci</name>
<affiliation wicri:level="1"><nlm:aff id="aff2"><institution>Department of Cognitive Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea></wicri:regionArea>
<wicri:regionArea># see nlm:aff region in country</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Krichmar, Jeffrey L" sort="Krichmar, Jeffrey L" uniqKey="Krichmar J" first="Jeffrey L." last="Krichmar">Jeffrey L. Krichmar</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><institution>Department of Computer Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea></wicri:regionArea>
<wicri:regionArea># see nlm:aff region in country</wicri:regionArea>
</affiliation>
<affiliation wicri:level="1"><nlm:aff id="aff2"><institution>Department of Cognitive Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea></wicri:regionArea>
<wicri:regionArea># see nlm:aff region in country</wicri:regionArea>
</affiliation>
</author>
</analytic>
<series><title level="j">Frontiers in Neurorobotics</title>
<idno type="eISSN">1662-5218</idno>
<imprint><date when="2015">2015</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en"><p>Neurorobots enable researchers to study how behaviors are produced by neural mechanisms in an uncertain, noisy, real-world environment. To investigate how the somatosensory system processes noisy, real-world touch inputs, we introduce a neurorobot called CARL-SJR, which has a full-body tactile sensory area. The design of CARL-SJR is such that it encourages people to communicate with it through gentle touch. CARL-SJR provides feedback to users by displaying bright colors on its surface. In the present study, we show that CARL-SJR is capable of learning associations between conditioned stimuli (CS; a color pattern on its surface) and unconditioned stimuli (US; a preferred touch pattern) by applying a spiking neural network (SNN) with neurobiologically inspired plasticity. Specifically, we modeled the primary somatosensory cortex, prefrontal cortex, striatum, and the insular cortex, which is important for hedonic touch, to process noisy data generated directly from CARL-SJR's tactile sensory area. To facilitate learning, we applied dopamine-modulated Spike Timing Dependent Plasticity (STDP) to our simulated prefrontal cortex, striatum, and insular cortex. To cope with noisy, varying inputs, the SNN was tuned to produce traveling waves of activity that carried spatiotemporal information. Despite the noisy tactile sensors, spike trains, and variations in subject hand swipes, the learning was quite robust. Further, insular cortex activities in the incremental pathway of dopaminergic reward system allowed us to control CARL-SJR's preference for touch direction without heavily pre-processed inputs. The emerged behaviors we found in this model match animal's behaviors wherein they prefer touch in particular areas and directions. Thus, the results in this paper could serve as an explanation on the underlying neural mechanisms for developing tactile preferences and hedonic touch.</p>
</div>
</front>
<back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Abraira, V E" uniqKey="Abraira V">V. E. Abraira</name>
</author>
<author><name sortKey="Ginty, D D" uniqKey="Ginty D">D. D. Ginty</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Benucci, A" uniqKey="Benucci A">A. Benucci</name>
</author>
<author><name sortKey="Frazor, R A" uniqKey="Frazor R">R. A. Frazor</name>
</author>
<author><name sortKey="Carandini, M" uniqKey="Carandini M">M. Carandini</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Bologna, L L" uniqKey="Bologna L">L. L. Bologna</name>
</author>
<author><name sortKey="Pinoteau, J" uniqKey="Pinoteau J">J. Pinoteau</name>
</author>
<author><name sortKey="Brasselet, R" uniqKey="Brasselet R">R. Brasselet</name>
</author>
<author><name sortKey="Maggiali, M" uniqKey="Maggiali M">M. Maggiali</name>
</author>
<author><name sortKey="Arleo, A" uniqKey="Arleo A">A. Arleo</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Bologna, L L" uniqKey="Bologna L">L. L. Bologna</name>
</author>
<author><name sortKey="Pinoteau, J" uniqKey="Pinoteau J">J. Pinoteau</name>
</author>
<author><name sortKey="Passot, J B" uniqKey="Passot J">J. B. Passot</name>
</author>
<author><name sortKey="Garrido, J A" uniqKey="Garrido J">J. A. Garrido</name>
</author>
<author><name sortKey="Vogel, J" uniqKey="Vogel J">J. Vogel</name>
</author>
<author><name sortKey="Vidal, E R" uniqKey="Vidal E">E. R. Vidal</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Boureau, Y L" uniqKey="Boureau Y">Y. L. Boureau</name>
</author>
<author><name sortKey="Dayan, P" uniqKey="Dayan P">P. Dayan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Breazeal, C" uniqKey="Breazeal C">C. Breazeal</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Brown, J" uniqKey="Brown J">J. Brown</name>
</author>
<author><name sortKey="Bullock, D" uniqKey="Bullock D">D. Bullock</name>
</author>
<author><name sortKey="Grossberg, S" uniqKey="Grossberg S">S. Grossberg</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Bucci, L D" uniqKey="Bucci L">L. D. Bucci</name>
</author>
<author><name sortKey="Chou, T S" uniqKey="Chou T">T.-S. Chou</name>
</author>
<author><name sortKey="Krichmar, J L" uniqKey="Krichmar J">J. L. Krichmar</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Buzsaki, G" uniqKey="Buzsaki G">G. Buzsaki</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Cannata, G" uniqKey="Cannata G">G. Cannata</name>
</author>
<author><name sortKey="Maggiali, M" uniqKey="Maggiali M">M. Maggiali</name>
</author>
<author><name sortKey="Metta, G" uniqKey="Metta G">G. Metta</name>
</author>
<author><name sortKey="Sandini, G" uniqKey="Sandini G">G. Sandini</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Caporale, N" uniqKey="Caporale N">N. Caporale</name>
</author>
<author><name sortKey="Dan, Y" uniqKey="Dan Y">Y. Dan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Carlson, K D" uniqKey="Carlson K">K. D. Carlson</name>
</author>
<author><name sortKey="Nageswaran, J M" uniqKey="Nageswaran J">J. M. Nageswaran</name>
</author>
<author><name sortKey="Dutt, N" uniqKey="Dutt N">N. Dutt</name>
</author>
<author><name sortKey="Krichmar, J L" uniqKey="Krichmar J">J. L. Krichmar</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Chen, Y" uniqKey="Chen Y">Y. Chen</name>
</author>
<author><name sortKey="Mckinstry, J L" uniqKey="Mckinstry J">J. L. McKinstry</name>
</author>
<author><name sortKey="Edelman, G M" uniqKey="Edelman G">G. M. Edelman</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Chorley, P" uniqKey="Chorley P">P. Chorley</name>
</author>
<author><name sortKey="Seth, A K" uniqKey="Seth A">A. K. Seth</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Craig, A D" uniqKey="Craig A">A. D. Craig</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Craig, A D" uniqKey="Craig A">A. D. Craig</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Craig, A D" uniqKey="Craig A">A. D. Craig</name>
</author>
<author><name sortKey="Bushnell, M C" uniqKey="Bushnell M">M. C. Bushnell</name>
</author>
<author><name sortKey="Zhang, E T" uniqKey="Zhang E">E. T. Zhang</name>
</author>
<author><name sortKey="Blomqvist, A" uniqKey="Blomqvist A">A. Blomqvist</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Dahiya, R S" uniqKey="Dahiya R">R. S. Dahiya</name>
</author>
<author><name sortKey="Metta, G" uniqKey="Metta G">G. Metta</name>
</author>
<author><name sortKey="Valle, M" uniqKey="Valle M">M. Valle</name>
</author>
<author><name sortKey="Sandini, G" uniqKey="Sandini G">G. Sandini</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Drew, P J" uniqKey="Drew P">P. J. Drew</name>
</author>
<author><name sortKey="Abbott, L F" uniqKey="Abbott L">L. F. Abbott</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Evans, M H" uniqKey="Evans M">M. H. Evans</name>
</author>
<author><name sortKey="Fox, C W" uniqKey="Fox C">C. W. Fox</name>
</author>
<author><name sortKey="Lepora, N F" uniqKey="Lepora N">N. F. Lepora</name>
</author>
<author><name sortKey="Pearson, M J" uniqKey="Pearson M">M. J. Pearson</name>
</author>
<author><name sortKey="Sullivan, J C" uniqKey="Sullivan J">J. C. Sullivan</name>
</author>
<author><name sortKey="Prescott, T J" uniqKey="Prescott T">T. J. Prescott</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Felleman, D J" uniqKey="Felleman D">D. J. Felleman</name>
</author>
<author><name sortKey="Van Essen, D C" uniqKey="Van Essen D">D. C. Van Essen</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Ferezou, I" uniqKey="Ferezou I">I. Ferezou</name>
</author>
<author><name sortKey="Haiss, F" uniqKey="Haiss F">F. Haiss</name>
</author>
<author><name sortKey="Gentet, L J" uniqKey="Gentet L">L. J. Gentet</name>
</author>
<author><name sortKey="Aronoff, R" uniqKey="Aronoff R">R. Aronoff</name>
</author>
<author><name sortKey="Weber, B" uniqKey="Weber B">B. Weber</name>
</author>
<author><name sortKey="Petersen, C C" uniqKey="Petersen C">C. C. Petersen</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Funahashi, S" uniqKey="Funahashi S">S. Funahashi</name>
</author>
<author><name sortKey="Bruce, C J" uniqKey="Bruce C">C. J. Bruce</name>
</author>
<author><name sortKey="Goldman Rakic, P S" uniqKey="Goldman Rakic P">P. S. Goldman-Rakic</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Gluck, M A" uniqKey="Gluck M">M. A. Gluck</name>
</author>
<author><name sortKey="Thompson, R F" uniqKey="Thompson R">R. F. Thompson</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Griffith, J S" uniqKey="Griffith J">J. S. Griffith</name>
</author>
<author><name sortKey="Horn, G" uniqKey="Horn G">G. Horn</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Han, F" uniqKey="Han F">F. Han</name>
</author>
<author><name sortKey="Caporale, N" uniqKey="Caporale N">N. Caporale</name>
</author>
<author><name sortKey="Dan, Y" uniqKey="Dan Y">Y. Dan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Houk, J C" uniqKey="Houk J">J. C. Houk</name>
</author>
<author><name sortKey="Adams, J L" uniqKey="Adams J">J. L. Adams</name>
</author>
<author><name sortKey="Barto, A G" uniqKey="Barto A">A. G. Barto</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Humphries, M D" uniqKey="Humphries M">M. D. Humphries</name>
</author>
<author><name sortKey="Lepora, N" uniqKey="Lepora N">N. Lepora</name>
</author>
<author><name sortKey="Wood, R" uniqKey="Wood R">R. Wood</name>
</author>
<author><name sortKey="Gurney, K" uniqKey="Gurney K">K. Gurney</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Izhikevich, E M" uniqKey="Izhikevich E">E. M. Izhikevich</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Izhikevich, E M" uniqKey="Izhikevich E">E. M. Izhikevich</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Izhikevich, E M" uniqKey="Izhikevich E">E. M. Izhikevich</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Izhikevich, E M" uniqKey="Izhikevich E">E. M. Izhikevich</name>
</author>
<author><name sortKey="Edelman, G M" uniqKey="Edelman G">G. M. Edelman</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Khamassi, M" uniqKey="Khamassi M">M. Khamassi</name>
</author>
<author><name sortKey="Humphries, M D" uniqKey="Humphries M">M. D. Humphries</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Khamassi, M" uniqKey="Khamassi M">M. Khamassi</name>
</author>
<author><name sortKey="Lallee, S" uniqKey="Lallee S">S. Lallee</name>
</author>
<author><name sortKey="Enel, P" uniqKey="Enel P">P. Enel</name>
</author>
<author><name sortKey="Procyk, E" uniqKey="Procyk E">E. Procyk</name>
</author>
<author><name sortKey="Dominey, P F" uniqKey="Dominey P">P. F. Dominey</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Koch, K W" uniqKey="Koch K">K. W. Koch</name>
</author>
<author><name sortKey="Fuster, J M" uniqKey="Fuster J">J. M. Fuster</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Krichmar, J L" uniqKey="Krichmar J">J. L. Krichmar</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Krichmar, J L" uniqKey="Krichmar J">J. L. Krichmar</name>
</author>
<author><name sortKey="Rohrbein, F" uniqKey="Rohrbein F">F. Rohrbein</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Li, Q" uniqKey="Li Q">Q. Li</name>
</author>
<author><name sortKey="Elbrechter, C" uniqKey="Elbrechter C">C. Elbrechter</name>
</author>
<author><name sortKey="Haschke, R" uniqKey="Haschke R">R. Haschke</name>
</author>
<author><name sortKey="Ritter, H" uniqKey="Ritter H">H. Ritter</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Ljungberg, T" uniqKey="Ljungberg T">T. Ljungberg</name>
</author>
<author><name sortKey="Apicella, P" uniqKey="Apicella P">P. Apicella</name>
</author>
<author><name sortKey="Schultz, W" uniqKey="Schultz W">W. Schultz</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Ljungberg, T" uniqKey="Ljungberg T">T. Ljungberg</name>
</author>
<author><name sortKey="Apicella, P" uniqKey="Apicella P">P. Apicella</name>
</author>
<author><name sortKey="Schultz, W" uniqKey="Schultz W">W. Schultz</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Lubenov, E V" uniqKey="Lubenov E">E. V. Lubenov</name>
</author>
<author><name sortKey="Siapas, A G" uniqKey="Siapas A">A. G. Siapas</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Maheshwari, V" uniqKey="Maheshwari V">V. Maheshwari</name>
</author>
<author><name sortKey="Saraf, R" uniqKey="Saraf R">R. Saraf</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Markram, H" uniqKey="Markram H">H. Markram</name>
</author>
<author><name sortKey="Gerstner, W" uniqKey="Gerstner W">W. Gerstner</name>
</author>
<author><name sortKey="Sjostrom, P J" uniqKey="Sjostrom P">P. J. Sjostrom</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Miyazaki, K W" uniqKey="Miyazaki K">K. W. Miyazaki</name>
</author>
<author><name sortKey="Miyazaki, K" uniqKey="Miyazaki K">K. Miyazaki</name>
</author>
<author><name sortKey="Doya, K" uniqKey="Doya K">K. Doya</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Montague, P R" uniqKey="Montague P">P. R. Montague</name>
</author>
<author><name sortKey="Dayan, P" uniqKey="Dayan P">P. Dayan</name>
</author>
<author><name sortKey="Sejnowski, T J" uniqKey="Sejnowski T">T. J. Sejnowski</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Morrison, I" uniqKey="Morrison I">I. Morrison</name>
</author>
<author><name sortKey="Bjornsdotter, M" uniqKey="Bjornsdotter M">M. Bjornsdotter</name>
</author>
<author><name sortKey="Olausson, H" uniqKey="Olausson H">H. Olausson</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Morrison, I" uniqKey="Morrison I">I. Morrison</name>
</author>
<author><name sortKey="Loken, L S" uniqKey="Loken L">L. S. Loken</name>
</author>
<author><name sortKey="Minde, J" uniqKey="Minde J">J. Minde</name>
</author>
<author><name sortKey="Wessberg, J" uniqKey="Wessberg J">J. Wessberg</name>
</author>
<author><name sortKey="Perini, I" uniqKey="Perini I">I. Perini</name>
</author>
<author><name sortKey="Nennesmo, I" uniqKey="Nennesmo I">I. Nennesmo</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="N Guyen, S" uniqKey="N Guyen S">S. N'Guyen</name>
</author>
<author><name sortKey="Pirim, P" uniqKey="Pirim P">P. Pirim</name>
</author>
<author><name sortKey="Meyer, J A" uniqKey="Meyer J">J.-A. Meyer</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Nageswaran, J M" uniqKey="Nageswaran J">J. M. Nageswaran</name>
</author>
<author><name sortKey="Dutt, N" uniqKey="Dutt N">N. Dutt</name>
</author>
<author><name sortKey="Krichmar, J L" uniqKey="Krichmar J">J. L. Krichmar</name>
</author>
<author><name sortKey="Nicolau, A" uniqKey="Nicolau A">A. Nicolau</name>
</author>
<author><name sortKey="Veidenbaum, A V" uniqKey="Veidenbaum A">A. V. Veidenbaum</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Nicola, S M" uniqKey="Nicola S">S. M. Nicola</name>
</author>
<author><name sortKey="Surmeier, J" uniqKey="Surmeier J">J. Surmeier</name>
</author>
<author><name sortKey="Malenka, R C" uniqKey="Malenka R">R. C. Malenka</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Olausson, H" uniqKey="Olausson H">H. Olausson</name>
</author>
<author><name sortKey="Wessberg, J" uniqKey="Wessberg J">J. Wessberg</name>
</author>
<author><name sortKey="Morrison, I" uniqKey="Morrison I">I. Morrison</name>
</author>
<author><name sortKey="Mcglone, F" uniqKey="Mcglone F">F. McGlone</name>
</author>
<author><name sortKey="Vallbo, A" uniqKey="Vallbo A">A. Vallbo</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Palmer, J H" uniqKey="Palmer J">J. H. Palmer</name>
</author>
<author><name sortKey="Gong, P" uniqKey="Gong P">P. Gong</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Pan, W X" uniqKey="Pan W">W. X. Pan</name>
</author>
<author><name sortKey="Schmidt, R" uniqKey="Schmidt R">R. Schmidt</name>
</author>
<author><name sortKey="Wickens, J R" uniqKey="Wickens J">J. R. Wickens</name>
</author>
<author><name sortKey="Hyland, B I" uniqKey="Hyland B">B. I. Hyland</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Pearson, M J" uniqKey="Pearson M">M. J. Pearson</name>
</author>
<author><name sortKey="Mitchinson, B" uniqKey="Mitchinson B">B. Mitchinson</name>
</author>
<author><name sortKey="Sullivan, J C" uniqKey="Sullivan J">J. C. Sullivan</name>
</author>
<author><name sortKey="Pipe, A G" uniqKey="Pipe A">A. G. Pipe</name>
</author>
<author><name sortKey="Prescott, T J" uniqKey="Prescott T">T. J. Prescott</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Rescorla, R A" uniqKey="Rescorla R">R. A. Rescorla</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Richert, M" uniqKey="Richert M">M. Richert</name>
</author>
<author><name sortKey="Nageswaran, J M" uniqKey="Nageswaran J">J. M. Nageswaran</name>
</author>
<author><name sortKey="Dutt, N" uniqKey="Dutt N">N. Dutt</name>
</author>
<author><name sortKey="Krichmar, J L" uniqKey="Krichmar J">J. L. Krichmar</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Rubino, D" uniqKey="Rubino D">D. Rubino</name>
</author>
<author><name sortKey="Robbins, K A" uniqKey="Robbins K">K. A. Robbins</name>
</author>
<author><name sortKey="Hatsopoulos, N G" uniqKey="Hatsopoulos N">N. G. Hatsopoulos</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Sato, T K" uniqKey="Sato T">T. K. Sato</name>
</author>
<author><name sortKey="Nauhaus, I" uniqKey="Nauhaus I">I. Nauhaus</name>
</author>
<author><name sortKey="Carandini, M" uniqKey="Carandini M">M. Carandini</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Scassellati, B" uniqKey="Scassellati B">B. Scassellati</name>
</author>
<author><name sortKey="Admoni, H" uniqKey="Admoni H">H. Admoni</name>
</author>
<author><name sortKey="Mataric, M" uniqKey="Mataric M">M. Mataric</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Schroeder, C L" uniqKey="Schroeder C">C. L. Schroeder</name>
</author>
<author><name sortKey="Hartmann, M J" uniqKey="Hartmann M">M. J. Hartmann</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Schultz, W" uniqKey="Schultz W">W. Schultz</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Schultz, W" uniqKey="Schultz W">W. Schultz</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Schultz, W" uniqKey="Schultz W">W. Schultz</name>
</author>
<author><name sortKey="Dayan, P" uniqKey="Dayan P">P. Dayan</name>
</author>
<author><name sortKey="Montague, P R" uniqKey="Montague P">P. R. Montague</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Seth, A K" uniqKey="Seth A">A. K. Seth</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Sewards, T V" uniqKey="Sewards T">T. V. Sewards</name>
</author>
<author><name sortKey="Sewards, M" uniqKey="Sewards M">M. Sewards</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Siegel, J Z" uniqKey="Siegel J">J. Z. Siegel</name>
</author>
<author><name sortKey="Crockett, M J" uniqKey="Crockett M">M. J. Crockett</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Silver, R A" uniqKey="Silver R">R. A. Silver</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Singer, T" uniqKey="Singer T">T. Singer</name>
</author>
<author><name sortKey="Critchley, H D" uniqKey="Critchley H">H. D. Critchley</name>
</author>
<author><name sortKey="Preuschoff, K" uniqKey="Preuschoff K">K. Preuschoff</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Soltoggio, A" uniqKey="Soltoggio A">A. Soltoggio</name>
</author>
<author><name sortKey="Lemme, A" uniqKey="Lemme A">A. Lemme</name>
</author>
<author><name sortKey="Reinhart, F" uniqKey="Reinhart F">F. Reinhart</name>
</author>
<author><name sortKey="Steil, J J" uniqKey="Steil J">J. J. Steil</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Soltoggio, A" uniqKey="Soltoggio A">A. Soltoggio</name>
</author>
<author><name sortKey="Steil, J J" uniqKey="Steil J">J. J. Steil</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Spigler, G" uniqKey="Spigler G">G. Spigler</name>
</author>
<author><name sortKey="Oddo, C M" uniqKey="Oddo C">C. M. Oddo</name>
</author>
<author><name sortKey="Carrozza, M C" uniqKey="Carrozza M">M. C. Carrozza</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Srinivasa, N" uniqKey="Srinivasa N">N. Srinivasa</name>
</author>
<author><name sortKey="Jiang, Q" uniqKey="Jiang Q">Q. Jiang</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Szatmary, B" uniqKey="Szatmary B">B. Szatmary</name>
</author>
<author><name sortKey="Izhikevich, E M" uniqKey="Izhikevich E">E. M. Izhikevich</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Tan, C O" uniqKey="Tan C">C. O. Tan</name>
</author>
<author><name sortKey="Bullock, D" uniqKey="Bullock D">D. Bullock</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Tops, M" uniqKey="Tops M">M. Tops</name>
</author>
<author><name sortKey="Russo, S" uniqKey="Russo S">S. Russo</name>
</author>
<author><name sortKey="Boksem, M A" uniqKey="Boksem M">M. A. Boksem</name>
</author>
<author><name sortKey="Tucker, D M" uniqKey="Tucker D">D. M. Tucker</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Williams, G V" uniqKey="Williams G">G. V. Williams</name>
</author>
<author><name sortKey="Castner, S A" uniqKey="Castner S">S. A. Castner</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wu, J Y" uniqKey="Wu J">J. Y. Wu</name>
</author>
<author><name sortKey="Xiaoying, H" uniqKey="Xiaoying H">H. Xiaoying</name>
</author>
<author><name sortKey="Chuan, Z" uniqKey="Chuan Z">Z. Chuan</name>
</author>
</analytic>
</biblStruct>
</listBibl>
</div1>
</back>
</TEI>
<pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">Front Neurorobot</journal-id>
<journal-id journal-id-type="iso-abbrev">Front Neurorobot</journal-id>
<journal-id journal-id-type="publisher-id">Front. Neurorobot.</journal-id>
<journal-title-group><journal-title>Frontiers in Neurorobotics</journal-title>
</journal-title-group>
<issn pub-type="epub">1662-5218</issn>
<publisher><publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta><article-id pub-id-type="pmid">26257639</article-id>
<article-id pub-id-type="pmc">4510776</article-id>
<article-id pub-id-type="doi">10.3389/fnbot.2015.00006</article-id>
<article-categories><subj-group subj-group-type="heading"><subject>Neuroscience</subject>
<subj-group><subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group><article-title>Learning touch preferences with a tactile robot using dopamine modulated STDP in a model of insular cortex</article-title>
</title-group>
<contrib-group><contrib contrib-type="author"><name><surname>Chou</surname>
<given-names>Ting-Shuo</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
<xref ref-type="author-notes" rid="fn001"><sup>*</sup>
</xref>
<uri xlink:type="simple" xlink:href="http://loop.frontiersin.org/people/177247/overview"></uri>
</contrib>
<contrib contrib-type="author"><name><surname>Bucci</surname>
<given-names>Liam D.</given-names>
</name>
<xref ref-type="aff" rid="aff2"><sup>2</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Krichmar</surname>
<given-names>Jeffrey L.</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2"><sup>2</sup>
</xref>
<uri xlink:type="simple" xlink:href="http://loop.frontiersin.org/people/473/overview"></uri>
</contrib>
</contrib-group>
<aff id="aff1"><sup>1</sup>
<institution>Department of Computer Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</aff>
<aff id="aff2"><sup>2</sup>
<institution>Department of Cognitive Sciences, University of California, Irvine</institution>
<country>Irvine, CA, USA</country>
</aff>
<author-notes><fn fn-type="edited-by"><p>Edited by: Andrea Soltoggio, Loughborough University, UK</p>
</fn>
<fn fn-type="edited-by"><p>Reviewed by: Mehdi Khamassi, Université Pierre et Marie Curie, France; Felix Reinhart, Bielefeld University, Germany</p>
</fn>
<corresp id="fn001">*Correspondence: Ting-Shuo Chou, Department of Cognitive Sciences, University of California, Irvine, 2224 Social & Behavioral Sciences Gateway, Irvine, CA 92697-5100, USA <email xlink:type="simple">tingshuc@uci.edu</email>
</corresp>
</author-notes>
<pub-date pub-type="epub"><day>22</day>
<month>7</month>
<year>2015</year>
</pub-date>
<pub-date pub-type="collection"><year>2015</year>
</pub-date>
<volume>9</volume>
<elocation-id>6</elocation-id>
<history><date date-type="received"><day>23</day>
<month>3</month>
<year>2015</year>
</date>
<date date-type="accepted"><day>02</day>
<month>7</month>
<year>2015</year>
</date>
</history>
<permissions><copyright-statement>Copyright © 2015 Chou, Bucci and Krichmar.</copyright-statement>
<copyright-year>2015</copyright-year>
<copyright-holder>Chou, Bucci and Krichmar</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/"><license-p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</license-p>
</license>
</permissions>
<abstract><p>Neurorobots enable researchers to study how behaviors are produced by neural mechanisms in an uncertain, noisy, real-world environment. To investigate how the somatosensory system processes noisy, real-world touch inputs, we introduce a neurorobot called CARL-SJR, which has a full-body tactile sensory area. The design of CARL-SJR is such that it encourages people to communicate with it through gentle touch. CARL-SJR provides feedback to users by displaying bright colors on its surface. In the present study, we show that CARL-SJR is capable of learning associations between conditioned stimuli (CS; a color pattern on its surface) and unconditioned stimuli (US; a preferred touch pattern) by applying a spiking neural network (SNN) with neurobiologically inspired plasticity. Specifically, we modeled the primary somatosensory cortex, prefrontal cortex, striatum, and the insular cortex, which is important for hedonic touch, to process noisy data generated directly from CARL-SJR's tactile sensory area. To facilitate learning, we applied dopamine-modulated Spike Timing Dependent Plasticity (STDP) to our simulated prefrontal cortex, striatum, and insular cortex. To cope with noisy, varying inputs, the SNN was tuned to produce traveling waves of activity that carried spatiotemporal information. Despite the noisy tactile sensors, spike trains, and variations in subject hand swipes, the learning was quite robust. Further, insular cortex activities in the incremental pathway of dopaminergic reward system allowed us to control CARL-SJR's preference for touch direction without heavily pre-processed inputs. The emerged behaviors we found in this model match animal's behaviors wherein they prefer touch in particular areas and directions. Thus, the results in this paper could serve as an explanation on the underlying neural mechanisms for developing tactile preferences and hedonic touch.</p>
</abstract>
<kwd-group><kwd>tactile robot</kwd>
<kwd>reinforcement learning</kwd>
<kwd>dopamine</kwd>
<kwd>STDP</kwd>
<kwd>insular cortex</kwd>
<kwd>somatosensory cortex</kwd>
</kwd-group>
<funding-group><award-group><funding-source id="cn001">NSF</funding-source>
<award-id rid="cn001">IIS-0910710</award-id>
</award-group>
<award-group><funding-source id="cn002">Qualcomm Technologies Incorporated</funding-source>
<award-id rid="cn002">THE-207651</award-id>
</award-group>
</funding-group>
<counts><fig-count count="10"></fig-count>
<table-count count="0"></table-count>
<equation-count count="13"></equation-count>
<ref-count count="77"></ref-count>
<page-count count="18"></page-count>
<word-count count="12936"></word-count>
</counts>
</article-meta>
</front>
<body><sec sec-type="intro" id="s1"><title>Introduction</title>
<p>Humans and other animals respond preferentially to different types of touches. For example most cats prefer to be petted from head to tail rather than the other way around. Although, tactile sensing is an active area of robotics research, which takes inspiration from biology and neuroscience, most tactile robots have been developed to sense the borders and shapes of objects (Pearson et al., <xref rid="B54" ref-type="bibr">2011</xref>
; Evans et al., <xref rid="B20" ref-type="bibr">2012</xref>
; Schroeder and Hartmann, <xref rid="B60" ref-type="bibr">2012</xref>
) or for grasping and detecting surfaces (Bologna et al., <xref rid="B3" ref-type="bibr">2011</xref>
, <xref rid="B4" ref-type="bibr">2013</xref>
; Spigler et al., <xref rid="B71" ref-type="bibr">2012</xref>
). The present paper introduces a tactile neurorobot that has a surface designed for petting. The robot has the ability to signal its preferences through coloration of its surface and auditory signals. We will use this neurorobot to explore neural mechanisms of learning in uncertain, real-world environments.</p>
<p>In a set of mutual reinforcement learning experiments, we demonstrate that a user can pair colors on the robot's surface with hand sweeps in preferred directions across the robot's surface. The spatiotemporal nature of tactile stimuli, as well as the noisy sensors and environments, in which they operate, make the perception of touch a complex problem. To address these issues, we introduce a biologically spiking neural network, which learns through a novel dopaminergic spike timing dependent plasticity mechanism. Specifically, the robot has built-in tactile preferences and a user must learn these preferences, as well as reward the robot by touching the robot in its preferred ways. Auditory tones were used to signal pleasure and disappointment. In this way, the robot can learn the association between a color and a gentle touch.</p>
<p>Because the neurorobot's main sensory modality was touch, we developed a neurobiologically plausible model of tactile sensing. Mammals have two tactile pathways; one which is fast and delivers fine touch resolution, and another which is slower with coarser resolution that delivers hedonic or value-laden touch information. It is this latter touch pathway that we will explore in the present experiments. In animals, sophisticated cutaneous mechanoreceptors in the skin are capable of perceiving temperature, indentation, stretch, vibration, and movement (Abraira and Ginty, <xref rid="B1" ref-type="bibr">2013</xref>
). Unlike primary visual cortex for visual information, primary somatosensory cortex (S1) is not the only first order cortical region processing tactile information from thalamus (TH). Insular cortex (IC), which was thought to be higher hierarchical cortical region receiving tactile information from secondary somatosensory cortex (S2) (Felleman and Van Essen, <xref rid="B21" ref-type="bibr">1991</xref>
), also processes tactile information ascending directly from posterior ventromedial thalamus (VMpo) in macaque (Sewards and Sewards, <xref rid="B65" ref-type="bibr">2002</xref>
) and human (Craig et al., <xref rid="B17" ref-type="bibr">1994</xref>
). The parallel pathways to insular cortex in mammals imply that a single piece of tactile information could be heterogeneously processed in different regions and then integrated in insular cortex. For instance, a gentle touch detected by mechanoreceptors with C-fiber and Aβ-fiber triggers spike trains going through TH→IC and TH→S1→S2→IC pathways, respectively. The spike trains invoke pleasant sensation, which is correlated with insular activity (Morrison et al., <xref rid="B46" ref-type="bibr">2011a</xref>
,<xref rid="B47" ref-type="bibr">b</xref>
). The pleasant sensation could be a state of emotional representation in anterior insular cortex (AIC) as a result of integrating tactile information along posterior insular (pIC), mid-insular (mIC) and anterior insular (AIC) (Craig, <xref rid="B15" ref-type="bibr">2002</xref>
, <xref rid="B16" ref-type="bibr">2009</xref>
). In the present study, we are interested in the neural mechanism for integrating tactile information in this area because the unconscious element of the pleasant sensation might link to the dopamine system (Schultz, <xref rid="B62" ref-type="bibr">2006</xref>
) and therefore defines innate preferences or values (Krichmar and Rohrbein, <xref rid="B37" ref-type="bibr">2013</xref>
).</p>
<p>To explore learning mechanisms for hedonic touch, we constructed a spiking neural network (SNN) model of the posterior insular cortex (pIC), somatosensory cortex, and the areas necessary for value-based learning. The neural dynamics in the model of pIC accounted for: (1) the robot's tactile preferences, (2) processing real-world tactile inputs with minimal pre-processing, (3) demonstrating that wave propagation is a viable means to generating precise spike timing in the face of noise, and (4) learning associations between neutral stimuli and hedonic touch.</p>
<p>Because hedonic touch requires a caresser and a caressee, we developed a human robot interaction study that required mutual reinforcement learning. To achieve these goals, we built a robot, named CARL-SJR (Cognitive Anteater Robotics Laboratory—Spike Judgment Robot), with a large tactile sensory area and a surface capable of displaying bright colors. CARL-SJR's behavior was controlled by the dual-pathway model (Brown et al., <xref rid="B7" ref-type="bibr">1999</xref>
; Tan and Bullock, <xref rid="B74" ref-type="bibr">2008</xref>
; Chorley and Seth, <xref rid="B14" ref-type="bibr">2011</xref>
), which minimizes prediction error signaled by dopamine (Schultz, <xref rid="B62" ref-type="bibr">2006</xref>
).</p>
</sec>
<sec sec-type="materials|methods" id="s2"><title>Materials and methods</title>
<sec><title>CARL-SJR neurorobot</title>
<p>The sensory encoding and learning experiments were conducted with a novel robot named CARL-SJR. CARL-SJR is autonomous, mobile, self-contained, and capable of tactile sensing and interaction (see Figure <xref ref-type="fig" rid="F1">1</xref>
). To give the robot a sense of touch, we incorporated an array of trackballs, which are typically found in cellphones and other devices. The trackball array can signal the direction and velocity of tactile stimuli. The robot's unique form factor encourages users to rub or pet its surface. The robot has LEDs co-located at each trackball, which can display a wide range of colors in response to touch. CARL-SJR has a large tactile sensory area and the ability to display bright colors on its shell (see Figure <xref ref-type="fig" rid="F1">1A</xref>
). CARL-SJR's shell has a 9-by-8 matrix of true color LEDs. Any animated color pattern can be programmed to display on its shell. The shell also has a 9-by-8 matrix of trackballs, which are used for sensing tactile input. The trackballs form a coordinate system with the upper left trackball mapped to the origin (0, 0), and the downward right trackball mapped to the maximum coordinates (8, 6) (see Figure <xref ref-type="fig" rid="F1">1B</xref>
). A trackball can detect a touch event in four directions (i.e., up, down, left, and right). Assuming a trackball rolls in the left direction, a sequence of touch events will be generated as shown by the timeline in Figure <xref ref-type="fig" rid="F1">1C</xref>
. In this example, the majority of touch events are in the left direction accompanied by noise in other directions. The current version of CARL-SJR mounted the tactile shell on an iRobot Create platform. A computer, which communicated with CARL-SJR over Bluetooth, executed the neural model, collected trackball data, controlled the LEDs on the shell, and controlled the motors and speakers on the iRobot Create. More details on the robot hardware can be found at Bucci et al. (<xref rid="B8" ref-type="bibr">2014</xref>
).</p>
<fig id="F1" position="float"><label>Figure 1</label>
<caption><p><bold>CARL-SJR is an interactive, tactile neurorobot. (A)</bold>
Photograph of CARL-SJR. The shell has a 9-by-8 matrix of trackballs each collocated with red, green, and blue color LEDs. Any animated color pattern is possible. <bold>(B)</bold>
The 9-by-8 matrix of trackballs forms the coordinate system of CARL-SJR. The most up left trackball is mapped to the origin (0, 0) while the most down right trackball is mapped to the coordinates (8, 6). <bold>(C)</bold>
A trackball can detect a touch event in four directions. Assuming a trackball rolls in the left direction, a sequence of touch events will be generated as shown by the timeline. The majority of touch events will be left events with events in other directions due to sensor noise. <bold>(D)</bold>
An example of an animated color pattern. The yellow pattern moves downward in 800 ms. <bold>(E)</bold>
An example of a typical touch pattern. The hand moves downward. Usually 4 ~ 7 trackballs are touched simultaneously. <bold>(F)</bold>
Schematic of the spiking neural network architecture that controlled CARL-SJR's behavior.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0001"></graphic>
</fig>
<p>CARL-SJR displayed color patterns as output, and took hand movements as input. The display patterns could be a solid color, mixed, or animated. Figure <xref ref-type="fig" rid="F1">1D</xref>
shows an example of an animated color pattern. The yellow pattern moves downward in 800 ms. Hand movements across the shell triggered touch events in the matrix of trackballs. Figure <xref ref-type="fig" rid="F1">1E</xref>
shows a typical touch pattern. The hand moves downward. Usually 4 ~ 7 trackballs are touched simultaneously.</p>
</sec>
<sec><title>Spiking neural network model</title>
<p>To support the present mutual reinforcement learning experiments, we built a spiking neural network (SNN) model using the large scale SNN simulator CARLsim to recognize tactile sensory input, and to control CARL-SJR's behavior (Nageswaran et al., <xref rid="B49" ref-type="bibr">2009</xref>
; Richert et al., <xref rid="B56" ref-type="bibr">2011</xref>
; Carlson et al., <xref rid="B12" ref-type="bibr">2014</xref>
). CARLsim was written in C/C++/CUDA and designed to leverage the parallel computing power of GPUs. The present SNN model had 13,000 neurons and 200,000 synapses and could run four times faster than real time. However, the model was slowed down to match the real time robotic application.</p>
<p>The SNN was designed to be biologically plausible and simulated somatosensory pathways, as well as neurally inspired learning. To support learning, we implemented a variation of the dual-pathway model (Brown et al., <xref rid="B7" ref-type="bibr">1999</xref>
; Tan and Bullock, <xref rid="B74" ref-type="bibr">2008</xref>
; Chorley and Seth, <xref rid="B14" ref-type="bibr">2011</xref>
), which minimizes prediction error signaled by dopamine (Schultz, <xref rid="B62" ref-type="bibr">2006</xref>
). In the present experiments, the Conditioned Stimulus (CS) was a color pattern displayed on CARL-SJR's surface, and the Unconditioned Stimulus (US) was a touch pattern initiated by the user sweeping his or her hand across CARL-SJR's surface. The decremental (dopamine) pathway for the CS (see PFC→STR→DA in Figure <xref ref-type="fig" rid="F1">1F</xref>
) decreased DA neurons' activity through inhibitory projections. In contrast, the incremental (dopamine) pathway for the US (see TH→pIC→DA in Figure <xref ref-type="fig" rid="F1">1F</xref>
) increased spikes of dopaminergic neurons. The complementary pathways converge on a group of DA neurons and control the DA response. Phasic neural activity in the incremental pathway for US might change the balance of excitation and inhibition, thus triggering a DA burst, which in turn signals striatum (STR) and PFC→STR synapses through dopaminergic projections. The decremental pathway is able to learn the timing of US and then increases inhibitory force on DA neurons for restoring the balance. The neural activities in prefrontal cortex (PFC) and striatum are crucial for learning the timing of US. Chorley and Seth's model incorporated pre-generated polychronous groups (Izhikevich, <xref rid="B30" ref-type="bibr">2006</xref>
) for precisely timed spikes in PFC. However, polychronous groups are a theoretical prediction and, to the best of our knowledge, have not been shown empirically. Moreover, it would be difficult to show repeatability of this precise timing in a computational model having noisy and uncertain inputs. Rather than relying on precisely timed polychrony or synfire chains, we used wavelike neural activity for propagating information through the simulated brain regions. These waves of neural activity have empirical support and do not require precisely timed spike sequences (Rubino et al., <xref rid="B57" ref-type="bibr">2006</xref>
; Benucci et al., <xref rid="B2" ref-type="bibr">2007</xref>
; Ferezou et al., <xref rid="B22" ref-type="bibr">2007</xref>
; Han et al., <xref rid="B26" ref-type="bibr">2008</xref>
; Wu et al., <xref rid="B77" ref-type="bibr">2008</xref>
; Lubenov and Siapas, <xref rid="B41" ref-type="bibr">2009</xref>
; Sato et al., <xref rid="B58" ref-type="bibr">2012</xref>
).</p>
<sec><title>Spiking neuron model</title>
<p>CARLsim incorporated a phenomenological model of a spiking neuron proposed by Izhikevich (<xref rid="B29" ref-type="bibr">2003</xref>
). The dynamics of each neuron is governed by the following equations:
<disp-formula id="E1"><label>(1)</label>
<mml:math id="M1"><mml:mrow><mml:mover accent="true"><mml:mi>v</mml:mi>
<mml:mo>˙</mml:mo>
</mml:mover>
<mml:mo>=</mml:mo>
<mml:mn>0.04</mml:mn>
<mml:msup><mml:mi>v</mml:mi>
<mml:mn>2</mml:mn>
</mml:msup>
<mml:mo>+</mml:mo>
<mml:mn>5</mml:mn>
<mml:mi>v</mml:mi>
<mml:mo>+</mml:mo>
<mml:mn>140</mml:mn>
<mml:mo>−</mml:mo>
<mml:mi>u</mml:mi>
<mml:mo>+</mml:mo>
<mml:mi>I</mml:mi>
</mml:mrow>
</mml:math>
</disp-formula>
<disp-formula id="E2"><label>(2)</label>
<mml:math id="M2"><mml:mover accent="true"><mml:mi>u</mml:mi>
<mml:mo>˙</mml:mo>
</mml:mover>
<mml:mo>=</mml:mo>
<mml:mi>a</mml:mi>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mi>b</mml:mi>
<mml:mi>v</mml:mi>
<mml:mo>−</mml:mo>
<mml:mi>u</mml:mi>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
</mml:math>
</disp-formula>
The variable <italic>v</italic>
is the membrane potential of a neuron and the variable <italic>u</italic>
is an abstract membrane recovery current. The variable <italic>I</italic>
is the input current (i.e., the current flow into a neuron). A neuron emits a spike if its membrane potential is higher than 30 mv and then resets according to the following equation:
<disp-formula id="E3"><label>(3)</label>
<mml:math id="M3"><mml:mi>i</mml:mi>
<mml:mi>f</mml:mi>
<mml:mtext> </mml:mtext>
<mml:mi>v</mml:mi>
<mml:mo>≥</mml:mo>
<mml:mn>30</mml:mn>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>t</mml:mi>
<mml:mi>h</mml:mi>
<mml:mi>e</mml:mi>
<mml:mi>n</mml:mi>
<mml:mtext> </mml:mtext>
<mml:mrow><mml:mo>{</mml:mo>
<mml:mrow><mml:mtable><mml:mtr><mml:mtd><mml:mrow><mml:mi>v</mml:mi>
<mml:mo>=</mml:mo>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mrow><mml:mi>u</mml:mi>
<mml:mo>=</mml:mo>
<mml:mi>u</mml:mi>
<mml:mo>+</mml:mo>
<mml:mi>d</mml:mi>
</mml:mrow>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:mrow>
</mml:mrow>
</mml:math>
</disp-formula>
Both excitatory regular spiking (RS) neurons and inhibitory fast spiking (FS) neurons were used in the model. For RS neurons, we set <italic>a</italic>
= 0.02, <italic>b</italic>
= 0.2, <italic>c</italic>
= −65.0, and <italic>d</italic>
= 8.0. For FS neurons, we set <italic>a</italic>
= 0.1, <italic>b</italic>
= 0.2, <italic>c</italic>
= −65.0, and <italic>d</italic>
= 2.0. For more biologically realistic dynamics, a conductance synapse model was used to calculate the input current for each neuron (Izhikevich and Edelman, <xref rid="B32" ref-type="bibr">2008</xref>
). The equation is:
<disp-formula id="E4"><label>(4)</label>
<mml:math id="M4"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:mi>I</mml:mi>
<mml:mo>=</mml:mo>
<mml:msub><mml:mi>g</mml:mi>
<mml:mrow><mml:mi>A</mml:mi>
<mml:mi>M</mml:mi>
<mml:mi>P</mml:mi>
<mml:mi>A</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mn>0</mml:mn>
<mml:mo>−</mml:mo>
<mml:mi>v</mml:mi>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mo>+</mml:mo>
<mml:msub><mml:mi>g</mml:mi>
<mml:mrow><mml:mi>N</mml:mi>
<mml:mi>M</mml:mi>
<mml:mi>D</mml:mi>
<mml:mi>A</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mfrac><mml:mrow><mml:msup><mml:mrow><mml:mrow><mml:mo>[</mml:mo>
<mml:mrow><mml:mfrac><mml:mrow><mml:mo>−</mml:mo>
<mml:mn>80</mml:mn>
<mml:mo>−</mml:mo>
<mml:mi>v</mml:mi>
</mml:mrow>
<mml:mrow><mml:mn>60</mml:mn>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
<mml:mo>]</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:msup>
</mml:mrow>
<mml:mrow><mml:mn>1</mml:mn>
<mml:mo>+</mml:mo>
<mml:msup><mml:mrow><mml:mrow><mml:mo>[</mml:mo>
<mml:mrow><mml:mfrac><mml:mrow><mml:mo>−</mml:mo>
<mml:mn>80</mml:mn>
<mml:mo>−</mml:mo>
<mml:mi>v</mml:mi>
</mml:mrow>
<mml:mrow><mml:mn>60</mml:mn>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
<mml:mo>]</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:msup>
</mml:mrow>
</mml:mfrac>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mn>0</mml:mn>
<mml:mo>−</mml:mo>
<mml:mi>v</mml:mi>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mo>+</mml:mo>
<mml:msub><mml:mi>g</mml:mi>
<mml:mrow><mml:mi>G</mml:mi>
<mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
<mml:msub><mml:mi>A</mml:mi>
<mml:mi>A</mml:mi>
</mml:msub>
</mml:mrow>
</mml:msub>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mo>−</mml:mo>
<mml:mn>70</mml:mn>
<mml:mo>−</mml:mo>
<mml:mi>v</mml:mi>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mo>+</mml:mo>
<mml:msub><mml:mi>g</mml:mi>
<mml:mrow><mml:mi>G</mml:mi>
<mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
<mml:msub><mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
</mml:msub>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">(</mml:mo>
<mml:mo>−</mml:mo>
<mml:mn>90</mml:mn>
<mml:mo>−</mml:mo>
<mml:mi>v</mml:mi>
<mml:mo stretchy="false">)</mml:mo>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
</disp-formula>
where <italic>v</italic>
is again the membrane potential and <italic>g</italic>
is the total conductance for ion channels created by different receptors (i.e., AMPA, NMDA, GABA<sub>A</sub>
, GABA<sub>B</sub>
). The conductance <italic>g</italic>
is increased by the amount of synaptic weight <italic>w</italic>
upon the arrival of a spike and decays along time as described by the equations below:
<disp-formula id="E5"><label>(5)</label>
<mml:math id="M5"><mml:msub><mml:mi>g</mml:mi>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>k</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mtext> </mml:mtext>
<mml:mo>=</mml:mo>
<mml:mstyle displaystyle="true"><mml:munderover><mml:mo>∑</mml:mo>
<mml:mi>j</mml:mi>
<mml:mi>N</mml:mi>
</mml:munderover>
<mml:mrow><mml:msub><mml:mi>w</mml:mi>
<mml:mrow><mml:mi>j</mml:mi>
<mml:mi>k</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mi>δ</mml:mi>
<mml:mo stretchy="false">(</mml:mo>
<mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>j</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mstyle>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>i</mml:mi>
<mml:mo>∈</mml:mo>
<mml:mo>{</mml:mo>
<mml:mi>A</mml:mi>
<mml:mi>M</mml:mi>
<mml:mi>P</mml:mi>
<mml:mi>A</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>N</mml:mi>
<mml:mi>M</mml:mi>
<mml:mi>D</mml:mi>
<mml:mi>A</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>G</mml:mi>
<mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
<mml:msub><mml:mi>A</mml:mi>
<mml:mi>A</mml:mi>
</mml:msub>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>G</mml:mi>
<mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
<mml:msub><mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
</mml:msub>
<mml:mo>}</mml:mo>
</mml:math>
</disp-formula>
<disp-formula id="E6"><label>(6)</label>
<mml:math id="M6"><mml:msub><mml:mover accent="true"><mml:mi>g</mml:mi>
<mml:mo>˙</mml:mo>
</mml:mover>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>k</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mo>−</mml:mo>
<mml:mfrac><mml:mrow><mml:msub><mml:mi>g</mml:mi>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>k</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mrow><mml:msub><mml:mi>τ</mml:mi>
<mml:mi>i</mml:mi>
</mml:msub>
</mml:mrow>
</mml:mfrac>
<mml:mtext> </mml:mtext>
<mml:mi>i</mml:mi>
<mml:mo>∈</mml:mo>
<mml:mo>{</mml:mo>
<mml:mi>A</mml:mi>
<mml:mi>M</mml:mi>
<mml:mi>P</mml:mi>
<mml:mi>A</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>N</mml:mi>
<mml:mi>M</mml:mi>
<mml:mi>D</mml:mi>
<mml:mi>A</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>G</mml:mi>
<mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
<mml:msub><mml:mi>A</mml:mi>
<mml:mi>A</mml:mi>
</mml:msub>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>G</mml:mi>
<mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
<mml:msub><mml:mi>A</mml:mi>
<mml:mi>B</mml:mi>
</mml:msub>
<mml:mo>}</mml:mo>
</mml:math>
</disp-formula>
where <italic>N</italic>
is the number of pre-synaptic neurons and <italic>w</italic>
<sub><italic>jk</italic>
</sub>
is the weight of the synapse connecting pre-synaptic neuron <italic>j</italic>
and post-synaptic neuron <italic>k</italic>
. δ is the Dirac delta function. <italic>t</italic>
is the current time (i.e., current simulation time step when we approximate the continuous function in discrete time steps) and <italic>t</italic>
<sub><italic>pre, j</italic>
</sub>
is the arrival time of the last spike from neuron <italic>j</italic>
. The decay constant τ<sub><italic>i</italic>
</sub>
was set to 5, 100, 6, and 150 ms for different receptors AMPA, NMDA, GABA<sub>A</sub>
, and GABA<sub>B</sub>
, respectively.</p>
</sec>
<sec><title>STDP, DA-STDP and DA-PSF</title>
<p>Spike timing dependent plasticity (STDP) (Caporale and Dan, <xref rid="B11" ref-type="bibr">2008</xref>
; Markram et al., <xref rid="B43" ref-type="bibr">2011</xref>
) was applied to excitatory and inhibitory synapses in the computational model. In our model, excitatory STDP and inhibitory STDP were used to develop and stabilize wavelike neural activity in the PFC area (see Section Wave Propagation in PFC). The synaptic weights were governed by the following equations:
<disp-formula id="E7"><label>(7)</label>
<mml:math id="M7"><mml:msub><mml:mover accent="true"><mml:mi>w</mml:mi>
<mml:mo>˙</mml:mo>
</mml:mover>
<mml:mrow><mml:mi>e</mml:mi>
<mml:mi>x</mml:mi>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:msup><mml:mi>A</mml:mi>
<mml:mo>+</mml:mo>
</mml:msup>
<mml:msup><mml:mi>e</mml:mi>
<mml:mrow><mml:mfrac><mml:mrow><mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mrow><mml:msub><mml:mi>τ</mml:mi>
<mml:mo>+</mml:mo>
</mml:msub>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:msup>
<mml:mi>δ</mml:mi>
<mml:mo stretchy="false">(</mml:mo>
<mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">)</mml:mo>
<mml:mo>−</mml:mo>
<mml:msup><mml:mi>A</mml:mi>
<mml:mo>−</mml:mo>
</mml:msup>
<mml:msup><mml:mi>e</mml:mi>
<mml:mrow><mml:mfrac><mml:mrow><mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mrow><mml:msub><mml:mi>τ</mml:mi>
<mml:mo>−</mml:mo>
</mml:msub>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:msup>
<mml:mi>δ</mml:mi>
<mml:mo stretchy="false">(</mml:mo>
<mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">)</mml:mo>
</mml:math>
</disp-formula>
<disp-formula id="E8"><label>(8)</label>
<mml:math id="M8"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:msub><mml:mover accent="true"><mml:mi>w</mml:mi>
<mml:mo>˙</mml:mo>
</mml:mover>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mi>n</mml:mi>
<mml:mi>h</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:msup><mml:mi>B</mml:mi>
<mml:mo>+</mml:mo>
</mml:msup>
<mml:msup><mml:mi>H</mml:mi>
<mml:mo>+</mml:mo>
</mml:msup>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mrow><mml:mo>|</mml:mo>
<mml:mrow><mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo>|</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mi>δ</mml:mi>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mo>+</mml:mo>
<mml:msup><mml:mi>B</mml:mi>
<mml:mo>+</mml:mo>
</mml:msup>
<mml:msup><mml:mi>H</mml:mi>
<mml:mo>+</mml:mo>
</mml:msup>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mrow><mml:mo>|</mml:mo>
<mml:mrow><mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo>|</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mi>δ</mml:mi>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mo>−</mml:mo>
<mml:msup><mml:mi>B</mml:mi>
<mml:mo>−</mml:mo>
</mml:msup>
<mml:msup><mml:mi>H</mml:mi>
<mml:mo>−</mml:mo>
</mml:msup>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mrow><mml:mo>|</mml:mo>
<mml:mrow><mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo>|</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mi>δ</mml:mi>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mo>−</mml:mo>
<mml:msup><mml:mi>B</mml:mi>
<mml:mo>−</mml:mo>
</mml:msup>
<mml:msup><mml:mi>H</mml:mi>
<mml:mo>−</mml:mo>
</mml:msup>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mrow><mml:mo>|</mml:mo>
<mml:mrow><mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo>|</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mi>δ</mml:mi>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
</disp-formula>
<disp-formula id="E9"><label>(9)</label>
<mml:math id="M9"><mml:msup><mml:mi>H</mml:mi>
<mml:mo>+</mml:mo>
</mml:msup>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mi>x</mml:mi>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mo>=</mml:mo>
<mml:mrow><mml:mo>{</mml:mo>
<mml:mrow><mml:mtable><mml:mtr><mml:mtd><mml:mrow><mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>i</mml:mi>
<mml:mi>f</mml:mi>
<mml:mtext> </mml:mtext>
<mml:mn>0</mml:mn>
<mml:mo><</mml:mo>
<mml:mi>x</mml:mi>
<mml:mo>≤</mml:mo>
<mml:mi>λ</mml:mi>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mrow><mml:mn>0</mml:mn>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>o</mml:mi>
<mml:mi>t</mml:mi>
<mml:mi>h</mml:mi>
<mml:mi>e</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>w</mml:mi>
<mml:mi>i</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:mrow>
</mml:mrow>
<mml:mtext></mml:mtext>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:msup><mml:mi>H</mml:mi>
<mml:mo>−</mml:mo>
</mml:msup>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mi>x</mml:mi>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mo>=</mml:mo>
<mml:mrow><mml:mo>{</mml:mo>
<mml:mrow><mml:mtable><mml:mtr><mml:mtd><mml:mrow><mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>i</mml:mi>
<mml:mi>f</mml:mi>
<mml:mtext> </mml:mtext>
<mml:mi>λ</mml:mi>
<mml:mo><</mml:mo>
<mml:mi>x</mml:mi>
<mml:mo>≤</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>γ</mml:mi>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mrow><mml:mn>0</mml:mn>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>o</mml:mi>
<mml:mi>t</mml:mi>
<mml:mi>h</mml:mi>
<mml:mi>e</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>w</mml:mi>
<mml:mi>i</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:mrow>
</mml:mrow>
</mml:math>
</disp-formula>
The variable <italic>w</italic>
<sub><italic>exc</italic>
</sub>
is an excitatory synaptic weight. <italic>t</italic>
<sub><italic>pre</italic>
</sub>
is the arrival time of last pre-synaptic spikes while <italic>t</italic>
<sub><italic>post</italic>
</sub>
is the time of post-synaptic spikes. δ is again the Dirac delta function. We set the E-STDP parameters <italic>A</italic>
<sup>+</sup>
<italic>/A</italic>
<sup>−</sup>
and τ<sub>+</sub>
/τ<sub>−</sub>
to 0.1/0.07 and 20/40 ms respectively. The variable <italic>w</italic>
<sub><italic>inh</italic>
</sub>
is an inhibitory synaptic weight. The I-STDP were modeled as a piecewise linear Mexican hat function as was described in Srinivasa and Jiang (<xref rid="B72" ref-type="bibr">2013</xref>
). The value of |<italic>t</italic>
<sub><italic>pre</italic>
</sub>
<italic>-t</italic>
<sub><italic>post</italic>
</sub>
|determines LTP or LTD. λ and γ define the ranges of inhibitory LTP and LTD. We set the I-STDP parameters <italic>B</italic>
<sup>+</sup>
<italic>/B</italic>
<sup>−</sup>
and λ/γ to 0.1/0.06 and 4/20 ms.</p>
<p>Dopamine modulated spike timing dependent plasticity (DA-STDP) served as the underlying neural mechanism for reinforcement learning and solving the distal reward problem (Izhikevich, <xref rid="B31" ref-type="bibr">2007</xref>
). Both the incremental and decremental pathways for CS receive dopamine signals (see Figure <xref ref-type="fig" rid="F1">1F</xref>
) and their synapses are subject to DA-STDP. In this form, the E-STDP function does not directly change synaptic weights, but instead modulates weights through an eligibility trace. The change of eligibility trace <italic>c</italic>
, dopamine value <italic>d</italic>
, and excitatory synaptic weight <italic>w</italic>
<sub><italic>exc</italic>
</sub>
are described by the following equations:
<disp-formula id="E10"><label>(10)</label>
<mml:math id="M10"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:mover accent="true"><mml:mi>c</mml:mi>
<mml:mo>˙</mml:mo>
</mml:mover>
<mml:mo>=</mml:mo>
<mml:mo>−</mml:mo>
<mml:mi>c</mml:mi>
<mml:mo>/</mml:mo>
<mml:msub><mml:mi>τ</mml:mi>
<mml:mi>c</mml:mi>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:msup><mml:mi>A</mml:mi>
<mml:mo>+</mml:mo>
</mml:msup>
<mml:msup><mml:mi>e</mml:mi>
<mml:mrow><mml:mfrac><mml:mrow><mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mrow><mml:msub><mml:mi>τ</mml:mi>
<mml:mo>+</mml:mo>
</mml:msub>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:msup>
<mml:mi>δ</mml:mi>
<mml:mo stretchy="false">(</mml:mo>
<mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">)</mml:mo>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mo>−</mml:mo>
<mml:msup><mml:mi>A</mml:mi>
<mml:mo>−</mml:mo>
</mml:msup>
<mml:msup><mml:mi>e</mml:mi>
<mml:mrow><mml:mfrac><mml:mrow><mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>o</mml:mi>
<mml:mi>s</mml:mi>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mrow><mml:msub><mml:mi>τ</mml:mi>
<mml:mo>−</mml:mo>
</mml:msub>
</mml:mrow>
</mml:mfrac>
</mml:mrow>
</mml:msup>
<mml:mi>δ</mml:mi>
<mml:mo stretchy="false">(</mml:mo>
<mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">)</mml:mo>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
</disp-formula>
<disp-formula id="E11"><label>(11)</label>
<mml:math id="M11"><mml:mover accent="true"><mml:mi>d</mml:mi>
<mml:mo>˙</mml:mo>
</mml:mover>
<mml:mo>=</mml:mo>
<mml:mo>−</mml:mo>
<mml:mi>d</mml:mi>
<mml:mo>/</mml:mo>
<mml:msub><mml:mi>τ</mml:mi>
<mml:mi>d</mml:mi>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:mi>d</mml:mi>
<mml:msub><mml:mi>a</mml:mi>
<mml:mrow><mml:mi>s</mml:mi>
<mml:mi>y</mml:mi>
<mml:mi>n</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mi>δ</mml:mi>
<mml:mo stretchy="false">(</mml:mo>
<mml:mi>t</mml:mi>
<mml:mo>−</mml:mo>
<mml:msub><mml:mi>t</mml:mi>
<mml:mrow><mml:mi>p</mml:mi>
<mml:mi>r</mml:mi>
<mml:mi>e</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="false">)</mml:mo>
</mml:math>
</disp-formula>
<disp-formula id="E12"><label>(12)</label>
<mml:math id="M12"><mml:msub><mml:mover accent="true"><mml:mi>w</mml:mi>
<mml:mo>˙</mml:mo>
</mml:mover>
<mml:mrow><mml:mi>e</mml:mi>
<mml:mi>x</mml:mi>
<mml:mi>c</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mi>c</mml:mi>
<mml:mi>d</mml:mi>
</mml:math>
</disp-formula>
The excitatory synaptic weight <italic>w</italic>
<sub><italic>exc</italic>
</sub>
is scaled by variable <italic>d</italic>
, which is the dopamine concentration of the target neural group (i.e., the post neural group projected by dopaminergic synapses). The dopamine value <italic>d</italic>
is increased by <italic>da</italic>
<sub><italic>syn</italic>
</sub>
, which is 0.04, for each spike reaching the target neural group. The value of <italic>d</italic>
ranges from the baseline value 1.0 μM to a peak value 20.0 μM. Please note, both <italic>d</italic>
and <italic>c</italic>
decay over time. The symbol δ represents the Dirac delta function and is one if there is an action potential at <italic>t</italic>
<sub><italic>pre</italic>
</sub>
and zero otherwise. The time constant τ<sub><italic>c</italic>
</sub>
and τ<sub><italic>d</italic>
</sub>
are 1000 and 50 ms as shown in Equations (10) and (11), respectively.</p>
<p>Dopamine modulated post-synaptic facilitation (DA-PSF) is the phenomenon where excitability of a post-synaptic neural group is modulated by dopamine (Nicola et al., <xref rid="B50" ref-type="bibr">2000</xref>
; Williams and Castner, <xref rid="B76" ref-type="bibr">2006</xref>
). A large portion of medium spiny neurons in striatum has D1 receptors to allow extra input current. We modeled this phenomenon as the following equation:
<disp-formula id="E13"><label>(13)</label>
<mml:math id="M13"><mml:msub><mml:mi>g</mml:mi>
<mml:mrow><mml:mi>e</mml:mi>
<mml:mi>f</mml:mi>
<mml:mi>c</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:msub><mml:mi>g</mml:mi>
<mml:mi>i</mml:mi>
</mml:msub>
<mml:mrow><mml:mo>(</mml:mo>
<mml:mrow><mml:mn>0.9</mml:mn>
<mml:mo>+</mml:mo>
<mml:mn>0.1</mml:mn>
<mml:mi>d</mml:mi>
</mml:mrow>
<mml:mo>)</mml:mo>
</mml:mrow>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>i</mml:mi>
<mml:mo>∈</mml:mo>
<mml:mo>{</mml:mo>
<mml:mi>A</mml:mi>
<mml:mi>M</mml:mi>
<mml:mi>P</mml:mi>
<mml:mi>A</mml:mi>
<mml:mo>,</mml:mo>
<mml:mtext> </mml:mtext>
<mml:mi>N</mml:mi>
<mml:mi>M</mml:mi>
<mml:mi>D</mml:mi>
<mml:mi>A</mml:mi>
<mml:mo>}</mml:mo>
</mml:math>
</disp-formula>
where <italic>d</italic>
is the dopamine value and <italic>g</italic>
<sub><italic>efc</italic>
</sub>
is the effective conductance used for calculating input current by Equation (4). Note that the minimum gain 1.0 is due to the baseline dopamine value of 1.0 μM.</p>
</sec>
<sec><title>Network architecture</title>
<p>The neural architecture for the present study, which is based on the dual-pathway model (Brown et al., <xref rid="B7" ref-type="bibr">1999</xref>
; Tan and Bullock, <xref rid="B74" ref-type="bibr">2008</xref>
; Chorley and Seth, <xref rid="B14" ref-type="bibr">2011</xref>
), is depicted in Figure <xref ref-type="fig" rid="F2">2A</xref>
. Color information, from CARL-SJR's shell, which represents the CS, is fed into both the decremental and incremental pathways. The incremental pathway also signals tactile information, which represents the US. It has been suggested that the incremental and decremental pathways balance each other out so that after learning the dopamine signal is suppressed when a reward is predicted by the CS and arrives at the expected time with a US.</p>
<fig id="F2" position="float"><label>Figure 2</label>
<caption><p><bold>Detailed Spiking Neural Network (SNN) architecture</bold>
. The SNN network model includes incremental (dopamine) pathway and decremental (dopamine) pathway. <bold>(A)</bold>
An overview of network architecture. The model consists of an Action Generator (AG) group, a sensory experience (EXP) group, prefrontal cortex (PFC), striatum (STR), an intermediate area (INT), thalamus (TH), thalamus/somatosensory area (TH:S1:S2), posterior insular cortex (pIC), mid-insular cortex (mIC), and a group of dopaminergic neurons (DA). <bold>(B)</bold>
Schematic of the projection patterns in the incremental pathway from AG to PFC to STR. <bold>(C)</bold>
Schematic of the projection patterns in the decremental pathway from TH:S1:S2 to pIC.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0002"></graphic>
</fig>
<p>The decremental pathway for the CS goes from the ActionGenerator (AG), to the prefrontal cortex (PFC), to the striatum (STR), and then to a pool of dopaminergic neurons (DA). All connections in the decremental pathway are excitatory, except for the STR→DA connections, which are inhibitory. The cortical area AG is used to generate spike trains that encode color responses. AG has 64 excitatory neurons for four CS input channels (i.e., red, green, blue, and yellow). Each neuron in a channel, which has 16 neurons totally, emits a spike if the corresponding CS input is presented. Figure <xref ref-type="fig" rid="F2">2B</xref>
illustrates more detailed connections of a channel from AG to PFC to STR. A pre-synaptic neuron in AG is topographically connected to a post-synaptic neuron in PFC based on the distance between neuron locations (see x_dist and y_dist in Figure <xref ref-type="fig" rid="F2">2B</xref>
). The smaller green circle in Figure <xref ref-type="fig" rid="F2">2C</xref>
defines the Excitatory Forward Projecting Radius (EFPR), which is the standard deviation of the Gaussian projection in spatial domain. Specifically, the distance between two neurons determines the connection probability (i.e., Gaussian distribution) and conductance delay (i.e., axonal delay). A channel in PFC has 2048 excitatory neurons and 512 inhibitory neurons. They are connected laterally according to Excitatory/Inhibitory Lateral Projecting Radius (ELPR/ILPR). The excitatory/inhibitory radius and synaptic weights were tuned to exhibit the behavior of traveling waves (Chen et al., <xref rid="B13" ref-type="bibr">2013</xref>
). If a CS input is presented, AG transmits spikes to the left side of the corresponding channel in PFC. The wide rectangular shape allows wavelike neural activity to propagate along the long side with precise timing. The idea behind the wave propagation in PFC is that the position of a neuron encodes the time relative to the release of a CS inputs (see Section Wave Propagation in PFC). PFC and STR are topologically connected according to EFPR as well. PFC→STR synaptic weights were subject to DA-STDP and sensitive to DA signals. The plastic PFC→STR synapses were able to learn (or predict) the timing of following US (see Section PFC-to-STR Synapses Learn to Predict the Timing of US). STR and DA are randomly connected and STR→DA synaptic weights were tuned to match the range of excitatory force from the incremental pathways.</p>
<p>The incremental pathway for the CS goes from AG to cortical area Expression (EXP), to Intermediate area (INT), and then to DA with excitatory projections. AG→EXP→INT→DA are randomly connected. EXP→INT connections are plastic synapses subject to DA-STDP and representations for CS inputs (i.e., red, green, blue, and yellow) will form in INT if a color pattern is reinforced (see Section CARL-SJR's Behaviors during Learning Multiple CS-US Pairs). INT→DA synaptic weights were tuned to match the range of inhibitory force from STR. INT was also linked to conditioned response (CR) in which CARL-SJR rotates its body if INT has more than 50 spikes within 50 ms.</p>
<p>For the incremental pathway, we implemented both the fast Aβ and the slower C-fiber tactile pathways. The incremental pathway for US starts with tactile inputs from touch events and projects from the parallel thalamus/thalamus:somatosensory cortex (TH/TH:S1:S2) paths, to the posterior insular cortex (pIC), to the medial insular cortex (mIC) and then to DA. The TH and TH:S1:S2 have four US input channels (i.e., up, down, left, and right). A touch event in Figure <xref ref-type="fig" rid="F1">1C</xref>
is one-to-one mapped to a spike in the corresponding channel in TH:S1:S2 where 9-by-8 neurons for each channel match the layout of trackballs. The spikes going through the TH:S1:S2→pIC pathway represent tactile information carried by the fast Aβ-fiber. To model fast spike transmission and acuteness in spatial resolution of this pathway, we topographically connected TH:S1:S2 and pIC as shown in Figure <xref ref-type="fig" rid="F2">2C</xref>
without any delay. In contrast, the spikes going through TH→pIC pathway represent tactile information carried by the slower C-fiber pathway, where a touch event is mapped to a period of tonic spikes. We delayed spike generation in TH by 700 ms to simulate the slower conduction speed of the C-fiber pathway, and fully connected TH to pIC to simulate poor spatial resolution. Since we hypothesize that a piece of tactile information is heterogeneously processed through the parallel paths and then integrated in pIC, the synaptic weights of TH:S1:S2→pIC and TH→pIC were tuned to fulfill the condition that neither TH:S1:S2→pIC nor TH→pIC can dominate the neural activity in pIC. Specifically, the neural activity in pIC was strong enough to drive neural response in mIC and then DA only when there was neural activity in both TH:S1:S2 and TH. As a result, the excitatory force on DA neurons reflects the integration of the neural activity in pIC. To suppress neural activity in pIC after mIC is signaled by pIC, we added feedback connections from mIC to inhibitory neurons in pIC to simulate the effect of shunting inhibition (Silver, <xref rid="B67" ref-type="bibr">2010</xref>
). The mIC is linked to the unconditioned response (UR), in which CARL-SJR sings a high tone if mIC has more than 30 spikes within 50 ms.</p>
<p>These complementary excitatory and inhibitory pathways converge on DA neurons. The interaction among STR, INT, and pIC activities controls DA response. The DA responded with a burst (see Section Control CARL-SJR's Tactile Preference through Insular Cortex Model) when the excitatory force was larger, a DA dip (see Section Extinguishing Behaviors after Learning) when the inhibitory force was larger, or spontaneous activity when excitatory and inhibitory forces were balanced. DA is connected to STR and INT through dopaminergic projections. The dopamine values of STR and INT modulate PFC→STR and EXP→INT synapses by DA-STDP and DA-PSF as indicated in Figure <xref ref-type="fig" rid="F2">2A</xref>
.</p>
<p>The synaptic weights, including initial, maximum, and minimum value, were tuned to match the number of pre-synaptic neurons and to prevent run-away neural dynamics. The conductance delays were tuned to maintain stable timing behaviors. The spontaneous firing rates in the cortical regions, ranging from 0.1 to 0.5 Hz (Griffith and Horn, <xref rid="B25" ref-type="bibr">1966</xref>
; Koch and Fuster, <xref rid="B35" ref-type="bibr">1989</xref>
), were tuned to support baseline neural activities, which are essential for STDP or DA-STDP. The complete network parameters are described in the Supplementary Materials.</p>
</sec>
</sec>
<sec><title>Tactile inputs</title>
<p>We analyzed the properties of tactile inputs (see Figure <xref ref-type="fig" rid="F3">3</xref>
) to gain insight into tactile processing. Figure <xref ref-type="fig" rid="F3">3A</xref>
shows raster plots and heat maps of three tactile inputs: (1) a downward hand sweep on CARL-SJR's left side, (2) a downward hand sweep in the middle, and (3) a rightward hand sweep in the middle. Since a touch event (see Figure <xref ref-type="fig" rid="F1">1C</xref>
) was a one-to-one mapping to spikes in TH:S1:S2 area, the raster plots shows touch events as well. We can see the timing of spikes is irregular and noisy. As for heat maps, they show which trackballs were touched, as well as a somatotopic map in TH:S1:S2 area. Because of the curvature of CARL-SJR's surface, the trackballs were not aligned along the x-axis but mostly aligned along the y-axis (see Figure <xref ref-type="fig" rid="F1">1B</xref>
). Thus, it was harder for users to touch complete rows of trackballs rather than columns of trackball.</p>
<fig id="F3" position="float"><label>Figure 3</label>
<caption><p><bold>Tactile stimulus analysis. (A)</bold>
Representative spike raster plots and heatmaps of three tactile inputs to the TH:S1:S2 area. On the left are raster plots where the x-axis represents times in milliseconds, and the y-axis represents the neuron number. The top row shows an upward hand sweep on the left side of CARL-SJR's shell, the middle row shows an upward hand sweep in the middle of CARL-SJR's shell, and bottom row shows a rightward hand sweep in the middle of CARL-SJR's shell. The raster plots illustrate how sensory input leads to irregular and noisy spike activity in TH:S1:S2. On the right are heatmaps that show the mean neural activity corresponding to the raster plots. The x and y position in the heat map reflects the somatotopic organization of the TH:S1:S2 neurons. <bold>(B)</bold>
The distribution of tactile inputs in contact duration, moving speed, number of events, and noise level (i.e., the amount of unexpected directional moves divided by the expected directional moves). The first row shows analysis based on 800 upward and downward moves while the second row shows analysis based on 600 leftward and rightward moves. The distributions of vertical and horizontal tactile input are quite different because locations of trackballs are asymmetric in vertical and horizontal direction.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0003"></graphic>
</fig>
<p>Figure <xref ref-type="fig" rid="F3">3B</xref>
shows distributions for touch duration, touch speed, number of touch events, and noise level over 1400 touch movements. The data of upward and downward movements were grouped in the first row while leftward and rightward movements were grouped in the second row because CARL-SJR's asymmetric trackball distribution affected the user's tactile inputs. The (mean, std) of touch durations were (935, 142) and (1,071, 128) ms for vertical and horizontal movement, respectively. The (mean, std) speeds were (7.3, 1.08) and (6.73, 0.93) inch/second for vertical and horizontal movement, respectively. The (mean, std) number of events were (207, 44) and (305, 69) for vertical and horizontal movement, respectively. The (mean, std) noise level were (8.31, 3.97%) and (19.72, 5.53%) for vertical and horizontal movement, respectively, where noise level was defined by the total number of direction events in an unexpected direction divided by the number of direction events in an expected direction. The noise level of horizontal movements is significantly higher due to CARL-SJR's curvature along y-axis.</p>
</sec>
<sec><title>Experimental paradigm</title>
<p>The experiment paradigm is shown in Figure <xref ref-type="fig" rid="F4">4</xref>
. A trial lasted 6 s. CARL-SJR initiated a trial by displaying a color on its surface (see Sections CARL-SJR's Behaviors during Learning Multiple CS-US Pairs and Extinguishing Behaviors after Learning for experimental details). A CS signal corresponding to the displayed color was input to the AG at 1.1 s after trial beginning. Effectively, neural activities were triggered in EXP (i.e., incremental pathway for CS) and PFC (i.e., decremental pathway) at 1.1 and 1.3 s respectively. It was the user's choice to reward CARL-SJR or not. If the user liked the displayed color, he/she could touch CARL-SJR within 2 s US window and the US signal (i.e., tactile input) was delivered to TH and TH:S1:S2, which in turn triggered a dopaminergic reward response (see Figure <xref ref-type="fig" rid="F2">2A</xref>
). Tactile input outside the US window did not activate this reward pathway. After the US window, there was a 2.9 s stabilizing period before the next trial. The experiment continued until learning achieved a desired level (e.g., the probability of CR is higher than UR).</p>
<fig id="F4" position="float"><label>Figure 4</label>
<caption><p><bold>The timing diagram illustrates the experimental setup</bold>
. Each trial lasts 6 s, where a CS signal is followed by a 2-s US window. It is a user's choice to provide a US signal (i.e., tactile input) or not to CARL-SJR. Tactile input outside the US window is unrewarded. After the US window has passed, there is a 2.9-s period before the next trial.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0004"></graphic>
</fig>
</sec>
</sec>
<sec sec-type="results" id="s3"><title>Results</title>
<sec><title>Control CARL-SJR's tactile preference through insular cortex model</title>
<p>To characterize the network dynamics in response to sensory input, we ran a set of simulation experiments to explore the SNN's ability to transfer tactile information across different simulated brain regions. The incremental pathway for US was implemented by the model of pIC, which exhibited complicated neural dynamics in response to hand movements. We set TH→pIC synaptic weights to 0.04 for each channel and ran simulations with 400 upward movements as inputs to TH:S1:S2 and TH. Figure <xref ref-type="fig" rid="F5">5A</xref>
illustrates representative neural activities in TH:S1:S2, TH, pIC, and DA. The raster plots in the first row show the spikes directly triggered by touch events in the upward direction. Green boxes indicate the tonic spikes generated by TH in the upward direction, which arrive at pIC 700 ms later because the conductance speed of C-fibers is slower than Aβ-fibers. Before the arrival of tonic spikes, pIC is at the state with low excitability. The spikes from TH:S1:S2 trigger local wavelike neural activity in pIC (see spikes earlier than green dashed lines in the second row). After the arrival of tonic spikes from TH, pIC transitions to a higher excitability state that can trigger global wavelike neural activities in the pIC (see spikes later than green dashed lines). Multiple waves may interact with others and may trigger a DA burst depending on the strength of the instant excitatory force (see histograms in the third row and first three columns for comparison). Note that, TH→pIC synaptic weights are crucial to the excitability of pIC, sustainability of global waves in pIC, and the probability of DA bursts.</p>
<fig id="F5" position="float"><label>Figure 5</label>
<caption><p><bold>Range of dopamine (DA) responses to varying input patterns (A) Five representative examples for DA response to tactile inputs</bold>
. (A1) strong DA burst. (A2) weak DA burst. (A3) no DA burst due to insufficient instantaneous activity at pIC area. (A4) no DA burst due to no integration of activity at TH and TH:S1:S2 area. (A5) multiple DA bursts. The first row shows raster plots of TH:S1:S2 and TH areas. The spikes of TH area are drawn as green shade regions. The second and third rows show the magnitude of activity at pIC area, which leads to different DA response. The fourth row shows DA response. <bold>(B)</bold>
TH→pIC synaptic weights affect DA response. (Left) The probability of a single DA burst based on synaptic weights and the moving direction of the tactile input. (Right) The probability of multiple DA bursts. Setting TH→pIC synaptic weight to 0.04 yields more than 50% of single DA burst while multiple DA burst is lower than 10%.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0005"></graphic>
</fig>
<p>The response of pIC was complex and influenced dopaminergic activity. Based on the DA response, we classified neural activities in pIC into five groups: (A1) strong DA burst, (A2) weak DA burst, (A3) no DA burst due to insufficient instant activity in pIC, (A4) no DA burst due to no global activity in pIC. (A5) multiple DA bursts. A gentle hand movement (with moderate speed) is most likely to trigger a DA burst. If the speed of a hand movement is too fast (e.g., the touch duration is less than 700 ms), the neural activity in TH:S1:S2 disappears before tonic spikes from TH tune pIC to excitable state and therefore pIC is unlikely to trigger a DA burst. On the other hand, if the speed is too slow, the excitatory force of TH:S1:S2 is too weak to trigger global waves in pIC even though pIC is tuned to excitable state by tonic spikes from TH.</p>
<p>We adjusted TH→pIC synaptic weights in different channels (i.e., up, down, left, and right) to control the probability of a DA burst linked to CARL-SJR's tactile preferences. To evaluate the probability of a DA burst against different TH→pIC synaptic weights, we ran simulations with 400 upward, 400 downward, 300 leftward, and 300 rightward movements as inputs to TH and TH:S1:S2. For each direction of inputs, we recorded the total number of single DA bursts and derived the burst probability (see Figure <xref ref-type="fig" rid="F5">5B</xref>
). To make CARL-SJR prefer rightward movements, for example, we set the TH→pIC synaptic weights in right channel to 0.04, and the other channels to 0.03. In this case, the probability of a DA burst was greater than 70% for rightward movements but less than 20% for other directions. As a result, CARL-SJR learned faster if the user gives rewards by rightward movements. Figure <xref ref-type="fig" rid="F5">5B</xref>
also shows how the asymmetric trackball distribution affects the number of touch events per hand movement. Note that there are typically more touch events for left and right movements than up and down movements (see the histograms at third column in Figure <xref ref-type="fig" rid="F3">3B</xref>
). This can have an effect on CARL-SJR's tactile preferences. For example, if we set TH→pIC synaptic weights to 0.04 for each channel, horizontal movements yield slightly higher probability of generating a DA burst than vertical movements (see the black dash line in Figure <xref ref-type="fig" rid="F5">5B</xref>
). Under the condition that TH→pIC synaptic weights for each channel are identical, CARL-SJR will prefer horizontal movements. In the experiments described below, the TH→pIC synaptic weights were set to 0.04 for the preferred direction, and 0.03 for the non-preferred direction. Taken together, these simulations showed that the simulated insular cortex could control dopamine bursts, which could lead to the shaping of tactile preferences.</p>
</sec>
<sec><title>Wave propagation in PFC</title>
<p>A critical requirement for learning is assigning the credit of a reward to the appropriate stimulus that occurred in the past. Wave propagation has been suggested to be important for computing this timing in classical conditioning tasks (Palmer and Gong, <xref rid="B52" ref-type="bibr">2014</xref>
). The idea is that wavelike neural activity in the cortex might encode timing information related to events.</p>
<p>To investigate if wave propagation was a viable mechanism for the spatiotemporal learning in the present experiments, we incorporated this idea into our simulated PFC and demonstrated traveling waves in a series of simulations (see Figure <xref ref-type="fig" rid="F6">6</xref>
). We modeled PFC as a long rectangular shape and tuned the conductance delay of lateral projections for excitatory neurons to be 15–20 ms and inhibitory neurons to be 1 ms. During a development period, we enabled E/I-STDP and delivered spikes to the most left side of PFC every 4 s. After 2000 s of development, the wave reliably propagated from the left side to the right side of PFC in 2 s as shown by the raster plot in Figure <xref ref-type="fig" rid="F6">6A</xref>
. The heat maps further show the location of a wave at a given time period. After starting the spike activity on the leftmost side of PFC, the wave reached the rightmost side of PFC at 1900 ms. The speed of waves is quite stable because of the lateral conductance delay. As a result, a neuron's location along x-axis encodes time information with high accuracy and reliability.</p>
<fig id="F6" position="float"><label>Figure 6</label>
<caption><p><bold>Time course of PFC wave propagating activity. (A)</bold>
Wave propagation in PFC area. (Left) The raster plot shows the time duration of the wave is around 2000 ms. (Right) the wave travels along x-axis of the PFC area and the locations of the wave at three time intervals, 0–100, 1000–1100, and 1800–1900 ms are shown from top to bottom, respectively. <bold>(B)</bold>
The fail rate of a wave and the ghost wave rate based on noise (Hz) and the height of PFC area (i.e., the number of PFC neurons along y-axis). The noise level affects the probability of a wave reaching from one end of PFC to another, and on generating a ghost wave. PFC area is more resistant to noise if the height is smaller than 12 neurons.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0006"></graphic>
</fig>
<p>Noise might stop wave propagation as well as trigger unexpected waves. To address this issue, we tested the robustness of wave propagation under a noisy environment (see Figure <xref ref-type="fig" rid="F6">6B</xref>
). Since we knew when a wave should arrive the most right side of PFC, we defined a fail case to be the absence of neural activity of the most right neurons [i.e., (508, y) ~ (511, y)] at the expected time window (i.e., 1950 ~ 2050 ms). We also defined a ghost wave case to be any occurrence of neural activity outside the expected time window. After the development period, the PFC was tested in 1000 trials where we delivered spikes to the most left neurons, which were connected to AG (see Figure <xref ref-type="fig" rid="F2">2B</xref>
), and then recorded the number of fail cases and ghost wave cases. We derived the fail rate (in percentage) and ghost wave rate against different magnitudes of noise (in Hz) and the height of PFC (in the number of neurons). The fail rate was under 10% if the height of PFC was 4 neurons, and was under 40% if the height of PFC was 8 neurons. However, the fail rate was around 80% when the height of PFC was more than 12 neurons. The magnitude of noise also affected ghost wave rate. The ghost wave rate reached 80% when the height of PFC was 8 neurons and the noise was 0.015 Hz. When the height of PFC was more than 12 PFC neurons, the ghost wave rates were greater than 100%, which meant there were more than one ghost waves in a trial. Both fail rate and ghost wave rate will affect learning efficiency of PFC→STR synapses (see Section PFC-to-STR Synapses Learn to Predict the Timing of US). Therefore, based on these analyses, we set the height of PFC to 4 neurons and set the noise to 0.01 Hz for stable wave propagation.</p>
</sec>
<sec><title>PFC-to-STR synapses learn to predict the timing of us</title>
<p>With the wave propagation mechanism in place, we still required a means to pair neutral stimuli (e.g., color) with innate value (e.g., a preferred touch). Therefore, we implemented dopamine modulated STDP and wave propagation in the network to associate a CS with a US with precise timing. The main function of the CS input coming from the PFC and STR in the decremental pathway is to predict the timing and strength of the ensuing US and to balance the excitatory and inhibitory forces on the DA neurons. Figure <xref ref-type="fig" rid="F7">7</xref>
shows simulation results that explain the underlying neural mechanisms. For each trial in the simulation, CS activated the PFC at 0 ms and triggered a propagating wave of activity. A DA burst was activated at 1100 ms to simulate the effect of US. We ran the simulation for 400 trials. The raster plots and histograms of STR activity for trial 1, trial 100, and trial 400 are shown on the left side of Figures <xref ref-type="fig" rid="F7">7A–C</xref>
, respectively. The PFC→STR weights were subject to DA-STDP. The PFC→STR synaptic weights for trial 1, trial 100, and trial 400 are shown on the right side of Figures <xref ref-type="fig" rid="F7">7A–C</xref>
, respectively. In trial 1, a burst in DA increases dopamine concentration in STR through dopaminergic projections around 1100 ms. Because the DA-PSF facilitated STR activity, STR neurons were further activated by pre-synaptic PFC neurons around 1100 ms. Since the CS triggered a wave propagating in PFC, there was always a small portion of PFC firing at any moment. Thus, the set of PFC neurons firing around 1100 ms caused their post-synaptic STR neurons to be potentiated. This neural mechanism led to a phasic neural activity around 1120 ms (due to axonal delay and latency of conductance based synapses) as shown in Figure <xref ref-type="fig" rid="F7">7A</xref>
. The higher dopamine concentration not only facilitated firings of STR neurons but also boosted the PFC→STR synaptic weights through DA-STDP. This process was repeated trial by trial. Over time, the set of PFC→STR synapses with the appropriate timing were strengthened. Comparing the weight maps in Figure <xref ref-type="fig" rid="F7">7</xref>
, a band of strong weights appeared at the PFC neurons encoding 1100 ms (those with x-coordinate from 280 to 300) in Figure <xref ref-type="fig" rid="F7">7B</xref>
and got stronger and earlier (i.e., shift toward the left) in Figure <xref ref-type="fig" rid="F7">7C</xref>
. To summarize, these strong synapses led to phasic neural activity in STR (see STR raster plots and histograms), which in turn suppressed the DA burst at the precise time through the decremental pathway (see DA raster plots).</p>
<fig id="F7" position="float"><label>Figure 7</label>
<caption><p><bold>PFC→STR connection learns the timing of dopaminergic bursts. (A)</bold>
Before learning, a DA burst will trigger transient STR activity through dopamine-modulated post-synaptic facilitation. The wave traveling at PFC area matches the STR activity and PFC→STR synaptic weights are reinforced by DA-STDP. The heatmaps on the right of the figure show the PFC→STR weights at different points, where the color represents the summation of the synaptic weights from a PFC neuron to a STR neuron. For instance, the PFC neuron at (290, 2) has 0.5 as the summation of its PFC→STR synaptic weights <bold>(B)</bold>
During learning, PFC→STR synaptic weights get stronger and the traveling wave at PFC area can actively trigger spikes at STR area, leading to a weaker DA burst. <bold>(C)</bold>
After learning, PFC→STR synaptic weights reach a maximum strength and the inhibitory force from STR area to DA area prevents a DA burst.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0007"></graphic>
</fig>
</sec>
<sec><title>CARL-SJR's behaviors during learning multiple CS-US pairs</title>
<p>The previous sections showed through simulation that the SNN could (1) encode tactile patterns, (2) encode timing through propagating waves, and (3) learn associations between neutral and value-laden stimuli. In this section, we show how these mechanisms can be used in real-world human robot interactive learning experiments.</p>
<p>We conducted the conditioning experiments described in Section Experimental Paradigm with different pairings of color and touch. In the first experiment (see Figure <xref ref-type="fig" rid="F8">8</xref>
), we set TH→pIC synaptic weights to 0.04 and CARL-SJR's display alternated between blue, and yellow. The user rubbed CARL-SJR in the downward direction whenever he/she saw the yellow pattern, and in the rightward direction when he/she saw the blue pattern. There were 160 trials for each color pattern, totally 320 trials. We recorded the DA responses for US and CS every trial and sampled the PFC→STR synaptic weights every 40 trials. The colored lines in Figure <xref ref-type="fig" rid="F8">8A</xref>
show the 75th percentile, median, and 25th percentile of DA response to the CS over 20 trials (totally 40 trials, 20 trials for each color pattern) while the gray lines show DA response to US. The trends for CS and US are clear; DA response shifted from US to CS for both color patterns as has been observed in empirical studies (Ljungberg et al., <xref rid="B40" ref-type="bibr">1992</xref>
; Schultz, <xref rid="B61" ref-type="bibr">1998</xref>
; Pan et al., <xref rid="B53" ref-type="bibr">2005</xref>
). Figure <xref ref-type="fig" rid="F8">8B</xref>
shows the average EXP→INT synaptic weights for each CS (i.e., red, green, blue, and yellow) during conditioning. Since the user only reinforced the blue and yellow patterns, the average synaptic weights directly reflect the user's preferential conditioning as shown by the higher values of blue and yellow EXP→INT weights. The PFC→STR weight maps of trial 40 and trial 320 are shown in Figure <xref ref-type="fig" rid="F8">8C</xref>
. These weight maps, which were driven by the user conditioning CARL-SJR, exhibit a strong group of weights associated with the CS with several bands of weights to a lesser degree. The width of a band indicates the imprecise timing of the US and the strength of a band indicates the probability of a US occurrence at the corresponding timing. The DA response to horizontal movements (i.e., US) decreased faster than vertical movements due to the stronger weights for the blue pattern. The weight maps generated by real time tactile inputs also demonstrated our approach can capture the timing and strength of US over a wide range and in a noisy, real-world environment. An interesting behavior is that CARL-SJR slightly prefers touches in the horizontal, rightward direction. In Section Control CARL-SJR's Tactile Preference through Insular Cortex Model, we showed that CARL-SJR by default prefer horizontal movements if we set TH→pIC synaptic weights to 0.04 for all channels. Because of this asymmetry, CARL-SJR shifted DA response to blue pattern faster than yellow pattern (see the higher blue median value in Figure <xref ref-type="fig" rid="F8">8A</xref>
and the higher blue line in Figure <xref ref-type="fig" rid="F8">8B</xref>
) by the default preference. The result here is also consistent with Section Control CARL-SJR's Tactile Preference through Insular Cortex Model.</p>
<fig id="F8" position="float"><label>Figure 8</label>
<caption><p><bold>CARL-SJR learned user preferences for color patterns. (A)</bold>
The colored lines show the 75th percentile, median, and 25th percentile of DA response to the CS over 20 trials (totally 40 trials, 20 trials for each color pattern) while the gray lines show DA response to US. <bold>(B)</bold>
The average INT→EXP synaptic weights over 20 trials (totally 40 trials, 20 trials for each color during conditioning. The user only reinforced the blue and yellow patterns. <bold>(C)</bold>
The PFC→STR weight maps of blue and yellow patterns over trial 40 and trial 320. The interpretation for the heatmap is the same as Figure <xref ref-type="fig" rid="F7">7</xref>
. <bold>(D)</bold>
The probability of CR and UR every five trials over five runs. The trends for CR and UR are clear and consistent with the DA spikes triggered by CS and US in <bold>(A)</bold>
.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0008"></graphic>
</fig>
<p>In a second set of experiments, we focused on the robot's behavior in the form of conditioned and unconditioned responses. For the conditioned response, which was based on INT activity, CARL-SJR rotated its body. For the unconditioned response, which was based on mIC activity, CARL-SJR emitted a high tone. The US was a downward movement, and the CS was the yellow pattern. We collected data for five runs and each run contained 120 trials. Each data point in Figure <xref ref-type="fig" rid="F8">8D</xref>
was calculated as the probability of CR and UR every five trials over five runs. The trends for CR and UR are clear and consistent with the DA spikes triggered by CS and US (see Figure <xref ref-type="fig" rid="F8">8A</xref>
). CARL-SJR learned to exhibit the CR with high probability after 40 trials and also suppressed the UR after around 70 trials. Taken together, these human-robot interaction experiments show that the proposed mechanisms can support learning in a real-world, noisy environment.</p>
</sec>
<sec><title>Extinguishing behaviors after learning</title>
<p>The ability to unlearn prior associations is critical for flexible behavior. In conditioning paradigms, omitting the US after learning can lead to extinction of the conditioned response. After conditioning, we repeated the experiment described in Section CARL-SJR's Behaviors during Learning Multiple CS-US Pairs without presenting the US in an additional 200 trials. The DA response and CARL-SJR's behavior were recorded as well. Figure <xref ref-type="fig" rid="F9">9A</xref>
shows a representative example for a dip of DA response due to the absence of expected rewards (Ljungberg et al., <xref rid="B39" ref-type="bibr">1991</xref>
). The raster plot of DA spikes shows the phasic neural activity around 1100 ms, which was triggered by CS via the incremental pathway. At 1300 ms, the CS arrived at PFC and triggered wave propagation. Based on the weight map developed during learning process, STR neurons exhibited a phasic neural activity around 2000 ms. Since no excitatory force was present around 2000 ms (due to absence of US), the inhibitory force from STR suppressed DA neurons and created a 400 ms interval without DA spikes. In Figure <xref ref-type="fig" rid="F9">9B</xref>
, we show the distribution of dip durations in 200 trials. In these experiments, if there was dip in DA activity longer than 400 ms, CARL-SJR emitted a low tone signaling the omission of an expected reward. During this experiment we observed CARL-SJR sang this unhappy low tone in 167 out of 200 trials, which is 83.5%.</p>
<fig id="F9" position="float"><label>Figure 9</label>
<caption><p><bold>Network response after learning when the US is omitted. (A)</bold>
A representative example for a dip in DA response due to the absence of expected rewards. The heatmap at the bottom shows the PFC→STR weights, where the color represents the summation of the synaptic weights from a PFC neuron to a STR neuron. <bold>(B)</bold>
Distribution of dip durations in 200 trials. The threshold for triggering a low tone is set to 400 ms. 83.5% trials trigger a low tone. <bold>(C)</bold>
The raster plot and histogram of 5 randomly selected DA neurons over 200 trials when the US is omitted.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0009"></graphic>
</fig>
<p>To emulate the <italic>in vivo</italic>
recordings from Ljungberg's experiments (Ljungberg et al., <xref rid="B39" ref-type="bibr">1991</xref>
), we randomly selected 5 out of 100 DA neurons over 200 trials. This emulated recording from a small sample of the available pool of dopaminergic neurons. From these recordings, we composed a raster plot and a histogram in Figure <xref ref-type="fig" rid="F9">9C</xref>
. The dip duration is roughly 500 ms (see light red bar in Figure <xref ref-type="fig" rid="F9">9C</xref>
) and there is no DA activity for around 100 ms (see deep red bar in Figure <xref ref-type="fig" rid="F9">9C</xref>
). These dip durations closely matches those reported in Ljungberg's study.</p>
<p>The DA dips, shown in Figure <xref ref-type="fig" rid="F9">9</xref>
, have been suggested to promote extinction when the reward associated with the US is absent. Therefore, we tested extinction behavior in a classical conditioning paradigm (see Figure <xref ref-type="fig" rid="F10">10</xref>
). After conditioning, we repeated the experiment without presenting the US. We then recorded the CR and UR for 5 runs and each run had 200 trials. We calculated the probability of the CR every 5 trials over 5 runs. As is indicative of extinction of a behavior, the probability of CR decayed to zero after 50–60 trials (see blue line in Figure <xref ref-type="fig" rid="F10">10</xref>
). We also plotted the average EXP→INT synaptic weights for the yellow pattern (see orange line in Figure <xref ref-type="fig" rid="F10">10</xref>
). The weights decayed and exhibited fluctuation around a steady level. The decay was due to dips of dopamine (see Figure <xref ref-type="fig" rid="F9">9B</xref>
) and noisy spontaneous firing activities in EXP and INT.</p>
<fig id="F10" position="float"><label>Figure 10</label>
<caption><p><bold>Extinction behavior when the US is omitted</bold>
. The blue line shows the average CR probability every 5 trials over 5 runs during the extinction trials. The probability of CR decayed to 0.0 after 50 trials, which is consistent with animal studies. The orange line shows the average EXP→INT synaptic weights for the reinforced color pattern. Note that the weights decay to a baseline level during extinction.</p>
</caption>
<graphic xlink:href="fnbot-09-00006-g0010"></graphic>
</fig>
</sec>
</sec>
<sec sec-type="discussion" id="s4"><title>Discussion</title>
<p>We introduced a tactile neurorobot, called CARL-SJR, which was capable of sensing noisy, real-world tactile inputs in a highly uncertain environment and taking tactile inputs with minimal pre-processing to convert touch events into spiking activity (see Section Control CARL-SJR's Tactile Preference through Insular Cortex Model). A detailed spiking neural network (SNN) model of somatosensory cortex and the insular cortex, which is known to be important for hedonic touch, drove CARL-SJR's behavior. Learning in the model was driven by a dual pathway model of dopaminergic learning and the emergence of traveling waves of neural activity that governed the release of dopamine and the timing of CS and US (see Sections Wave Propagation in PFC and PFC-to-STR Synapses Learn to Predict the Timing of US). CARL-SJR demonstrated the ability to associate multiple CS's (i.e., color patterns displayed on its shell) with US's (i.e., user hand sweeps across its shell). For example, in Section CARL-SJR's Behaviors during Learning Multiple CS-US Pairs, we showed that the model supported associations between blue-right and yellow-down during a training session. Moreover, the model was able to learn despite trial-by-trial variations in CS-US intervals due to the uncertainties of user inputs.</p>
<p>The human-robot interaction studies with CARL-SJR exhibits the paradigm of mutual reinforced learning. Specifically, the robot can learn the user's preferences through conditioning tasks while the user can learn the robot's tactile preference through the robot's responses. CARL-SJR's style of human-robot interaction may be applications in socially assistive robotics (Scassellati et al., <xref rid="B59" ref-type="bibr">2012</xref>
) or socially affective robots (Breazeal, <xref rid="B6" ref-type="bibr">2009</xref>
). Compared to other social robots, CARL-SJR is somewhat unique in that it focuses on tactile rather than visual interaction.</p>
<p>CARL-SJR was designed to encourage interaction through touch. Tactile sensing is an active area of robotics research that takes inspiration from biology and neuroscience. For example, inspired by rodents and other mammals with vibrissae, whiskered robots have been developed to sense the borders and shape of objects (N'Guyen et al., <xref rid="B48" ref-type="bibr">2010</xref>
; Pearson et al., <xref rid="B54" ref-type="bibr">2011</xref>
; Evans et al., <xref rid="B20" ref-type="bibr">2012</xref>
; Schroeder and Hartmann, <xref rid="B60" ref-type="bibr">2012</xref>
). Fingers and hands have been developed for humanoid robots to enable grasping and detecting surfaces (Bologna et al., <xref rid="B3" ref-type="bibr">2011</xref>
, <xref rid="B4" ref-type="bibr">2013</xref>
; Spigler et al., <xref rid="B71" ref-type="bibr">2012</xref>
; Li et al., <xref rid="B38" ref-type="bibr">2013</xref>
). Most of these robots are constructed from custom-made materials and sensing circuits for touch (Sewards and Sewards, <xref rid="B65" ref-type="bibr">2002</xref>
; Cannata et al., <xref rid="B10" ref-type="bibr">2008</xref>
; Maheshwari and Saraf, <xref rid="B42" ref-type="bibr">2008</xref>
; Dahiya et al., <xref rid="B18" ref-type="bibr">2010</xref>
). Creating an artificial tactile system is difficult for many reasons. For example, the sensors must cover a large range and be compliant with the surfaces with which they interact. Moreover, the spatiotemporal nature of tactile stimuli, as well as the noisy sensors and environments in which they operate, make the perception of touch a complex problem. To provide a large surface that could handle a wide range of user inputs, we utilized a matrix of trackballs, which are found in many cellphones, across CARL-SJR's curved shell. The size, shape, and resolution was a good fit for the types of social interactions for which CARL-SJR was designed (Bucci et al., <xref rid="B8" ref-type="bibr">2014</xref>
).</p>
<p>Driven by the dual-pathway model, the acquisition and extinction behaviors of CARL-SJR are consistent with animal behavioral studies (Rescorla, <xref rid="B55" ref-type="bibr">1988</xref>
). Further, the neural activities in our model are consistent with <italic>in vivo</italic>
neural recordings as well. First, the robot has built-in tactile preferences. This matches the animals' behaviors wherein they prefer social touches in particular areas and directions. The neural mechanism for integrating tactile information in pIC area could serve as an explanation to animals' innate tactile preferences. Second, assuming the user knows the tactile preferences in advance or learns them, the user can try to reward the robot by touching the robot in its preferred ways. In the present experiments, the unconditioned response (UR) was mapped to mIC activity resulting in a high tone signaled by the robot. The high tone was used as feedback for the user to know that CARL-SJR enjoyed this touch. Third, the robot can learn the association between a color and a gentle touch. In the present paper, a color displaying on the robot's surface, which was spontaneously generated, was considered the conditioned stimulus (CS), and a gentle touch in an innate preferred direction (i.e., unexpected reward) served as the unconditioned stimulus (US). After learning, the robot associated the reinforced color, which was facilitated by DA, with the future reward, which is a preferred touch. Moreover, the DA response shifted from US to CS, as has been observed experimentally (Ljungberg et al., <xref rid="B40" ref-type="bibr">1992</xref>
; Schultz, <xref rid="B61" ref-type="bibr">1998</xref>
; Pan et al., <xref rid="B53" ref-type="bibr">2005</xref>
). CARL-SJR expressed its conditioned response (CR) when INT activity was high and signaled the CR with a spinning motion and bright colors displayed on its surface. Fourth, the robot was depressed if a gentle touch was expected, but not given. This occurred as a dip in DA activity when the expected US was withheld. We mapped the duration of a dip to a low tone, which sounded unhappy. This behavior was triggered by a dip of dopamine concentration level (Ljungberg et al., <xref rid="B39" ref-type="bibr">1991</xref>
).</p>
<sec><title>Comparison with previous computational models</title>
<p>To associate temporally separated events (i.e., CS and US), many computational models assumed the firing activities of neurons responding to an earlier event slowly decay and the sustained firing activities are associated neurons with the ensuing US event through STDP (Gluck and Thompson, <xref rid="B24" ref-type="bibr">1987</xref>
; Drew and Abbott, <xref rid="B19" ref-type="bibr">2006</xref>
). Similarly, the slowly decaying eligibility trace (Houk et al., <xref rid="B27" ref-type="bibr">1995</xref>
) has been applied to associate temporally separated events in dopamine modulated STDP (Izhikevich, <xref rid="B31" ref-type="bibr">2007</xref>
). The idea of slowly decaying eligibility trace was also successfully applied to rate-based neurons where Soltoggio and Steil used rare neural correlations to calculate the eligibility trace (Soltoggio and Steil, <xref rid="B70" ref-type="bibr">2013</xref>
). This approach was further validated on iCub for classical and operant conditioning tasks (Soltoggio et al., <xref rid="B69" ref-type="bibr">2013</xref>
). Chorley and Seth later integrated the DA-STDP mechanism into the dual-pathway model (Chorley and Seth, <xref rid="B14" ref-type="bibr">2011</xref>
). Their model successfully accounted for a wide range of reward-related DA responses. A different approach to conditioning paradigms is to incorporate temporally separated events as propagating spiking waves and associate these events through the spatiotemporal interaction of these waves (Palmer and Gong, <xref rid="B52" ref-type="bibr">2014</xref>
).</p>
<p>Using wave propagation to solve the temporal credit assignment problem has some interesting features that address limitations in other neurobiologically plausible reinforcement learning rules. The early dopamine model of reinforcement learning was very similar to temporal difference learning (Montague et al., <xref rid="B45" ref-type="bibr">1996</xref>
; Schultz et al., <xref rid="B63" ref-type="bibr">1997</xref>
). From empirical data and computational modeling, dopamine appeared to track the reward prediction error. However, in the model, the dopamine signal moved backward in time in successive trials until it corresponded to the stimulus that was predictive of reward. This movement of a dopamine signal over time has not been observed empirically. Others have proposed an eligibility trace as a means to solve the credit assignment problem (Izhikevich, <xref rid="B31" ref-type="bibr">2007</xref>
; Soltoggio and Steil, <xref rid="B70" ref-type="bibr">2013</xref>
). However, since the amplitude of the trace from the time of the CS to the time of the US can be quite small, it requires many trials to make a strong association. Moreover, there is little empirical support for a biological process to support this type of learning that lasts over many seconds. In contrast, wave propagation has empirical support and does not have the limitations described above (Rubino et al., <xref rid="B57" ref-type="bibr">2006</xref>
; Benucci et al., <xref rid="B2" ref-type="bibr">2007</xref>
; Ferezou et al., <xref rid="B22" ref-type="bibr">2007</xref>
; Han et al., <xref rid="B26" ref-type="bibr">2008</xref>
; Wu et al., <xref rid="B77" ref-type="bibr">2008</xref>
; Lubenov and Siapas, <xref rid="B41" ref-type="bibr">2009</xref>
; Sato et al., <xref rid="B58" ref-type="bibr">2012</xref>
). These waves have the appropriate timescale, are robust to variations in timing, and can send a strong enough signal to support associative learning in a plausible number of trials.</p>
<p>Another interesting approach to solving the credit assignment was proposed by Khamassi et al. (<xref rid="B34" ref-type="bibr">2011</xref>
), where dopamine neurons produced a reward prediction error signal in response to any salient event and affected synaptic plasticity when it co-occurred with a motor efference property. This would also address the limitations described above. However, in our present experiments we do not have a motor efference copy. The CS is a color display on the robot's shell and does not produce a motor action. The motor command is actually from the subject interacting with the robot. Still, this may be an interesting approach to implement in future models.</p>
<p>CARL-SJR's SNN model, which integrated the slow C-fiber and fast Aβ-fiber tactile pathways to the pIC, Chorley and Seth's dual-pathway model, and Palmer's spiking wave propagation, demonstrated associative learning in the real-world with a robot receiving noisy user input. Several prerequisites or neural behaviors in Chorley's and Palmer's work limit their real-world applications. Palmer's model successfully shifts the neural response from US to CS. However, the US could trigger both a CR and a UR at the same time if the CS is not presented before US. PFC activity in Chorley and Seth's model was implemented with a pre-generated polychronous group (Izhikevich, <xref rid="B30" ref-type="bibr">2006</xref>
). The appearance of consistent polychronous groups in a noisy environment is difficult. For example, the criteria for re-occurrence of a polychronous group was set to a low threshold (i.e., 25% neurons of a group) in Szatmary and Izhikevich (<xref rid="B73" ref-type="bibr">2010</xref>
). In this case, only a small portion of neurons showed time-locked spike patterns. In contrast, when the criteria for re-occurrence was set to a high threshold (e.g., 100% neurons of a group) as in Bucci et al. (<xref rid="B8" ref-type="bibr">2014</xref>
), the polychronous groups were very small, rarely occurred, and lasted for less than 40 ms. In this prior neurorobot study, these polychronous groups did not match the PFC activity in Chorley's model where time-locked cortical spike patterns sustained for 1 s. Both Chorley's and our model exhibit an interesting neural mechanism in which the STR response was a little bit later than the DA response in early trials (see Figure <xref ref-type="fig" rid="F7">7A</xref>
) and then shifted backward to match the timing of the DA response in late trials (see Figure <xref ref-type="fig" rid="F7">7C</xref>
). Whereas Chorley and Seth did not discuss how the PFC→STR synaptic weights may affect the learning progress, we showed the PFC→STR weight maps were indicative of the US timing and this may support temporal associations.</p>
<p>Our present PFC model incorporated the idea of spiking wave propagation in Palmer's model, which made pre-generated spikes or pre-processing of the CS unnecessary. The location of a neuron in PFC and the topographical projections to STR encoded the time relative to the CS in the decremental pathway. The PFC→STR weight map clearly reflected the learning status (see Figure <xref ref-type="fig" rid="F7">7</xref>
). The firing rate of our PFC model is consistent with experimental studies: <0.5 Hz in the resting state (Koch and Fuster, <xref rid="B35" ref-type="bibr">1989</xref>
) and 5–40 Hz when behaving (Funahashi et al., <xref rid="B23" ref-type="bibr">1989</xref>
). Further, our model has been validated to handle the uncertainty of CS-US interval within 2 s while Chorley reported the valid US window to be (500 ± 100 ms). We also used inhibitory fast spiking neurons to simulate striatal medium spiny neurons (Humphries et al., <xref rid="B28" ref-type="bibr">2009</xref>
). The characteristic short spiking burst of a FS neuron facilitated learning in the decremental pathway. The FS neuron transitioned to an excitable state due to DA-PSF, and activity from a pre-synaptic RS neuron in PFC triggered a spike train in the post-synaptic FS STR neuron. This resulted in multiple increases in the synaptic weights through LTP. This scenario was dependent on increased dopamine activity.</p>
<p>The model guiding CARL-SJR's behavior was also unique in that it implemented the separate and parallel pathways for transmitting touch information to the cortex, in which fine touch is well represented in the somatosensory cortex, and hedonic caressing appears to represented in the insular cortex (Sewards and Sewards, <xref rid="B65" ref-type="bibr">2002</xref>
; Olausson et al., <xref rid="B51" ref-type="bibr">2010</xref>
; Morrison et al., <xref rid="B46" ref-type="bibr">2011a</xref>
). The response of the fast Aβ-fiber tactile pathway is reflected in the fine resolution somatotopic response shown in Figure <xref ref-type="fig" rid="F3">3A</xref>
. Less is known about insular neuronal activity in response to touch. However, it has been suggested that the insular cortex can make predictions of internal states (Singer et al., <xref rid="B68" ref-type="bibr">2009</xref>
; Seth, <xref rid="B64" ref-type="bibr">2013</xref>
) and the neural architecture driving CARL-SJR's pleasure seeking behavior supports these claims (see Figure <xref ref-type="fig" rid="F2">2</xref>
).</p>
</sec>
<sec><title>CARL-SJR's limitations</title>
<p>CARL-SJR has several limitations. We developed a PFC wave with relatively low internal noise (i.e., spontaneous neural activity). The maximum noise in Figure <xref ref-type="fig" rid="F6">6B</xref>
is 0.015 Hz, which is 10 times smaller than the spontaneous firing activity (i.e., white noise) in resting PFC suggested by the experimental study of Koch and Fuster (<xref rid="B35" ref-type="bibr">1989</xref>
). The low spontaneous firing activity extremely slows down the extinction process in the conditioning task (see Section Extinguishing Behaviors after Learning) because the probability to decorrelate coupled neurons in PFC and STR is too low. We left this issue to a future improvement on developing wave propagation under a typical noise level.</p>
<p>The peak dopamine value in our model is 20 μM, which is much higher than 3 μM reported in Izhikevich's and Chorley's models. The range of dopamine value was tuned to result in adequate learning rates in the neurorobot experiments. We could lower the dopamine value and keep the observable learning speed if some compensatory neural mechanisms were implemented. For example, a replay mechanism for the CS and US pairings when CARL-SJR “sleeps” (Buzsaki, <xref rid="B9" ref-type="bibr">1998</xref>
). However, to incorporate a true replay mechanism would require substantial efforts in building a hippocampus model for offline learning (Khamassi and Humphries, <xref rid="B33" ref-type="bibr">2012</xref>
). A simple approach, which would be effective in the present architecture, would be to simulate replay by injecting the CS and US into the model when CARL-SJR is not actively behaving.</p>
<p>The neural response of mechanoreceptors with C-fiber is tuned for gentle speeds (Morrison et al., <xref rid="B46" ref-type="bibr">2011a</xref>
). We did not capture this characteristic because the trackballs, as they are currently designed, cannot detect speed locally (i.e., the number of touch events is not proportional to the rolling speed of a trackball). We could calculate the speed of a movement across multiple trackballs. However, this approach requires substantial pre-processing and therefore, contradicts our design choice.</p>
</sec>
</sec>
<sec sec-type="conclusions" id="s5"><title>Conclusions</title>
<p>CARL-SJR is a neurorobot whose behavior is guided by a SNN model of tactile pathways in the cortex, and demonstrates user defined entraining of a robot through touch. By incorporating dopamine modulated learning with traveling waves of neural activity, we have shown a biologically plausible method of instrumental conditioning. Our SNN model can be easily extended for robotic applications in reinforcement learning paradigms. Future directions include dopaminergic projections to the frontal cortex and implementation of the serotoninergic (5-HT) system (Krichmar, <xref rid="B36" ref-type="bibr">2013</xref>
). DA is linked to rewards and curiosity-seeking behavior and 5-HT may be linked to risk aversion and withdrawn behavior (Tops et al., <xref rid="B75" ref-type="bibr">2009</xref>
; Boureau and Dayan, <xref rid="B5" ref-type="bibr">2011</xref>
; Siegel and Crockett, <xref rid="B66" ref-type="bibr">2013</xref>
). It has also been suggested that 5-HT plays a role in waiting for a delayed reward (Miyazaki et al., <xref rid="B44" ref-type="bibr">2012</xref>
). By combining the DA and 5-HT systems, we could make CARL-SJR not only learn through rewards but also cost. Moreover, following the notion of 5-HT being related temporal discounting, we could use 5-HT levels to modulate CARL-SJR's impulsiveness. These additions would make CARL-SJR's behavior more interesting and such a system could have applications in the fields of socially assistive and socially affective robotics.</p>
<sec><title>Conflict of interest statement</title>
<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
</sec>
</body>
<back><ack><p>This research was supported by NSF under Grant IIS-0910710 and by a subcontract (THE-207651) from Qualcomm Technologies Incorporated.</p>
</ack>
<sec sec-type="supplementary-material" id="s6"><title>Supplementary material</title>
<p>The Supplementary Material for this article can be found online at: <ext-link ext-link-type="uri" xlink:href="http://journal.frontiersin.org/article/10.3389/fnbot.2015.00006">http://journal.frontiersin.org/article/10.3389/fnbot.2015.00006</ext-link>
</p>
<supplementary-material content-type="local-data"><media xlink:href="DataSheet1.PDF"><caption><p>Click here for additional data file.</p>
</caption>
</media>
</supplementary-material>
</sec>
<ref-list><title>References</title>
<ref id="B1"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Abraira</surname>
<given-names>V. E.</given-names>
</name>
<name><surname>Ginty</surname>
<given-names>D. D.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>The sensory neurons of touch</article-title>
. <source>Neuron</source>
<volume>79</volume>
, <fpage>618</fpage>
–<lpage>639</lpage>
. <pub-id pub-id-type="doi">10.1016/j.neuron.2013.07.051</pub-id>
<pub-id pub-id-type="pmid">23972592</pub-id>
</mixed-citation>
</ref>
<ref id="B2"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Benucci</surname>
<given-names>A.</given-names>
</name>
<name><surname>Frazor</surname>
<given-names>R. A.</given-names>
</name>
<name><surname>Carandini</surname>
<given-names>M.</given-names>
</name>
</person-group>
(<year>2007</year>
). <article-title>Standing waves and traveling waves distinguish two circuits in visual cortex</article-title>
. <source>Neuron</source>
<volume>55</volume>
, <fpage>103</fpage>
–<lpage>117</lpage>
. <pub-id pub-id-type="doi">10.1016/j.neuron.2007.06.017</pub-id>
<pub-id pub-id-type="pmid">17610820</pub-id>
</mixed-citation>
</ref>
<ref id="B3"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bologna</surname>
<given-names>L. L.</given-names>
</name>
<name><surname>Pinoteau</surname>
<given-names>J.</given-names>
</name>
<name><surname>Brasselet</surname>
<given-names>R.</given-names>
</name>
<name><surname>Maggiali</surname>
<given-names>M.</given-names>
</name>
<name><surname>Arleo</surname>
<given-names>A.</given-names>
</name>
</person-group>
(<year>2011</year>
). <article-title>Encoding/decoding of first and second order tactile afferents in a neurorobotic application</article-title>
. <source>J. Physiol. Paris</source>
<volume>105</volume>
, <fpage>25</fpage>
–<lpage>35</lpage>
. <pub-id pub-id-type="doi">10.1016/j.jphysparis.2011.08.002</pub-id>
<pub-id pub-id-type="pmid">21911056</pub-id>
</mixed-citation>
</ref>
<ref id="B4"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bologna</surname>
<given-names>L. L.</given-names>
</name>
<name><surname>Pinoteau</surname>
<given-names>J.</given-names>
</name>
<name><surname>Passot</surname>
<given-names>J. B.</given-names>
</name>
<name><surname>Garrido</surname>
<given-names>J. A.</given-names>
</name>
<name><surname>Vogel</surname>
<given-names>J.</given-names>
</name>
<name><surname>Vidal</surname>
<given-names>E. R.</given-names>
</name>
<etal></etal>
</person-group>
. (<year>2013</year>
). <article-title>A closed-loop neurobotic system for fine touch sensing</article-title>
. <source>J. Neural Eng</source>
. <volume>10</volume>
:<fpage>046019</fpage>
. <pub-id pub-id-type="doi">10.1088/1741-2560/10/4/046019</pub-id>
<pub-id pub-id-type="pmid">23883543</pub-id>
</mixed-citation>
</ref>
<ref id="B5"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Boureau</surname>
<given-names>Y. L.</given-names>
</name>
<name><surname>Dayan</surname>
<given-names>P.</given-names>
</name>
</person-group>
(<year>2011</year>
). <article-title>Opponency revisited: competition and cooperation between dopamine and serotonin</article-title>
. <source>Neuropsychopharmacology</source>
<volume>36</volume>
, <fpage>74</fpage>
–<lpage>97</lpage>
. <pub-id pub-id-type="doi">10.1038/npp.2010.151</pub-id>
<pub-id pub-id-type="pmid">20881948</pub-id>
</mixed-citation>
</ref>
<ref id="B6"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Breazeal</surname>
<given-names>C.</given-names>
</name>
</person-group>
(<year>2009</year>
). <article-title>Role of expressive behaviour for robots that learn from people</article-title>
. <source>Philos. Trans. R. Soc. Lond. B Biol. Sci</source>
. <volume>364</volume>
, <fpage>3527</fpage>
–<lpage>3538</lpage>
. <pub-id pub-id-type="doi">10.1098/rstb.2009.0157</pub-id>
<pub-id pub-id-type="pmid">19884147</pub-id>
</mixed-citation>
</ref>
<ref id="B7"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Brown</surname>
<given-names>J.</given-names>
</name>
<name><surname>Bullock</surname>
<given-names>D.</given-names>
</name>
<name><surname>Grossberg</surname>
<given-names>S.</given-names>
</name>
</person-group>
(<year>1999</year>
). <article-title>How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues</article-title>
. <source>J. Neurosci</source>
. <volume>19</volume>
, <fpage>10502</fpage>
–<lpage>10511</lpage>
. <pub-id pub-id-type="pmid">10575046</pub-id>
</mixed-citation>
</ref>
<ref id="B8"><mixed-citation publication-type="book"><person-group person-group-type="author"><name><surname>Bucci</surname>
<given-names>L. D.</given-names>
</name>
<name><surname>Chou</surname>
<given-names>T.-S.</given-names>
</name>
<name><surname>Krichmar</surname>
<given-names>J. L.</given-names>
</name>
</person-group>
(<year>2014</year>
). <article-title>Sensory decoding in a tactile, interactive neurorobot</article-title>
, in <source>Paper Presented at the Robotics and Automation (ICRA), 2014 IEEE International Conference on</source>
(<publisher-loc>Hong Kong</publisher-loc>
).</mixed-citation>
</ref>
<ref id="B9"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Buzsaki</surname>
<given-names>G.</given-names>
</name>
</person-group>
(<year>1998</year>
). Memory consolidation during sleep: a neurophysiological perspective. <source>J. Sleep Res</source>
. <volume>7</volume>
(<issue>Suppl. 1</issue>
), <fpage>17</fpage>
–<lpage>23</lpage>
. <pub-id pub-id-type="pmid">9682189</pub-id>
</mixed-citation>
</ref>
<ref id="B10"><mixed-citation publication-type="book"><person-group person-group-type="author"><name><surname>Cannata</surname>
<given-names>G.</given-names>
</name>
<name><surname>Maggiali</surname>
<given-names>M.</given-names>
</name>
<name><surname>Metta</surname>
<given-names>G.</given-names>
</name>
<name><surname>Sandini</surname>
<given-names>G.</given-names>
</name>
</person-group>
(<year>2008</year>
). <article-title>An embedded artificial skin for humanoid robots</article-title>
, in <source>Paper Presented at the Multisensor Fusion and Integration for Intelligent Systems, 2008. MFI 2008. IEEE International Conference on</source>
(<publisher-loc>Seoul</publisher-loc>
).</mixed-citation>
</ref>
<ref id="B11"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Caporale</surname>
<given-names>N.</given-names>
</name>
<name><surname>Dan</surname>
<given-names>Y.</given-names>
</name>
</person-group>
(<year>2008</year>
). <article-title>Spike timing-dependent plasticity: a Hebbian learning rule</article-title>
. <source>Annu. Rev. Neurosci</source>
. <volume>31</volume>
, <fpage>25</fpage>
–<lpage>46</lpage>
. <pub-id pub-id-type="doi">10.1146/annurev.neuro.31.060407.125639</pub-id>
<pub-id pub-id-type="pmid">18275283</pub-id>
</mixed-citation>
</ref>
<ref id="B12"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Carlson</surname>
<given-names>K. D.</given-names>
</name>
<name><surname>Nageswaran</surname>
<given-names>J. M.</given-names>
</name>
<name><surname>Dutt</surname>
<given-names>N.</given-names>
</name>
<name><surname>Krichmar</surname>
<given-names>J. L.</given-names>
</name>
</person-group>
(<year>2014</year>
). <article-title>An efficient automated parameter tuning framework for spiking neural networks</article-title>
. <source>Front. Neurosci</source>
. <volume>8</volume>
:<issue>10</issue>
. <pub-id pub-id-type="doi">10.3389/fnins.2014.00010</pub-id>
<pub-id pub-id-type="pmid">24550771</pub-id>
</mixed-citation>
</ref>
<ref id="B13"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Chen</surname>
<given-names>Y.</given-names>
</name>
<name><surname>McKinstry</surname>
<given-names>J. L.</given-names>
</name>
<name><surname>Edelman</surname>
<given-names>G. M.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>Versatile networks of simulated spiking neurons displaying winner-take-all behavior</article-title>
. <source>Front. Comput. Neurosci</source>
. <volume>7</volume>
:<issue>16</issue>
. <pub-id pub-id-type="doi">10.3389/fncom.2013.00016</pub-id>
<pub-id pub-id-type="pmid">23515493</pub-id>
</mixed-citation>
</ref>
<ref id="B14"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Chorley</surname>
<given-names>P.</given-names>
</name>
<name><surname>Seth</surname>
<given-names>A. K.</given-names>
</name>
</person-group>
(<year>2011</year>
). <article-title>Dopamine-signaled reward predictions generated by competitive excitation and inhibition in a spiking neural network model</article-title>
. <source>Front. Comput. Neurosci</source>
. <volume>5</volume>
:<issue>21</issue>
. <pub-id pub-id-type="doi">10.3389/fncom.2011.00021</pub-id>
<pub-id pub-id-type="pmid">21629770</pub-id>
</mixed-citation>
</ref>
<ref id="B15"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Craig</surname>
<given-names>A. D.</given-names>
</name>
</person-group>
(<year>2002</year>
). <article-title>How do you feel? Interoception: the sense of the physiological condition of the body</article-title>
. <source>Nat. Rev. Neurosci</source>
. <volume>3</volume>
, <fpage>655</fpage>
–<lpage>666</lpage>
. <pub-id pub-id-type="doi">10.1038/nrn894</pub-id>
<pub-id pub-id-type="pmid">12154366</pub-id>
</mixed-citation>
</ref>
<ref id="B16"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Craig</surname>
<given-names>A. D.</given-names>
</name>
</person-group>
(<year>2009</year>
). <article-title>How do you feel—now? The anterior insula and human awareness</article-title>
. <source>Nat. Rev. Neurosci</source>
. <volume>10</volume>
, <fpage>59</fpage>
–<lpage>70</lpage>
. <pub-id pub-id-type="doi">10.1038/nrn2555</pub-id>
<pub-id pub-id-type="pmid">19096369</pub-id>
</mixed-citation>
</ref>
<ref id="B17"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Craig</surname>
<given-names>A. D.</given-names>
</name>
<name><surname>Bushnell</surname>
<given-names>M. C.</given-names>
</name>
<name><surname>Zhang</surname>
<given-names>E. T.</given-names>
</name>
<name><surname>Blomqvist</surname>
<given-names>A.</given-names>
</name>
</person-group>
(<year>1994</year>
). <article-title>A thalamic nucleus specific for pain and temperature sensation</article-title>
. <source>Nature</source>
<volume>372</volume>
, <fpage>770</fpage>
–<lpage>773</lpage>
. <pub-id pub-id-type="doi">10.1038/372770a0</pub-id>
<pub-id pub-id-type="pmid">7695716</pub-id>
</mixed-citation>
</ref>
<ref id="B18"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dahiya</surname>
<given-names>R. S.</given-names>
</name>
<name><surname>Metta</surname>
<given-names>G.</given-names>
</name>
<name><surname>Valle</surname>
<given-names>M.</given-names>
</name>
<name><surname>Sandini</surname>
<given-names>G.</given-names>
</name>
</person-group>
(<year>2010</year>
). <article-title>Tactile sensing—from humans to humanoids</article-title>
. <source>Robotics IEEE Trans</source>
. <volume>26</volume>
, <fpage>1</fpage>
–<lpage>20</lpage>
. <pub-id pub-id-type="doi">10.1109/TRO.2009.2033627</pub-id>
</mixed-citation>
</ref>
<ref id="B19"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Drew</surname>
<given-names>P. J.</given-names>
</name>
<name><surname>Abbott</surname>
<given-names>L. F.</given-names>
</name>
</person-group>
(<year>2006</year>
). <article-title>Extending the effects of spike-timing-dependent plasticity to behavioral timescales</article-title>
. <source>Proc. Natl. Acad. Sci. U.S.A</source>
. <volume>103</volume>
, <fpage>8876</fpage>
–<lpage>8881</lpage>
. <pub-id pub-id-type="doi">10.1073/pnas.0600676103</pub-id>
<pub-id pub-id-type="pmid">16731625</pub-id>
</mixed-citation>
</ref>
<ref id="B20"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Evans</surname>
<given-names>M. H.</given-names>
</name>
<name><surname>Fox</surname>
<given-names>C. W.</given-names>
</name>
<name><surname>Lepora</surname>
<given-names>N. F.</given-names>
</name>
<name><surname>Pearson</surname>
<given-names>M. J.</given-names>
</name>
<name><surname>Sullivan</surname>
<given-names>J. C.</given-names>
</name>
<name><surname>Prescott</surname>
<given-names>T. J.</given-names>
</name>
</person-group>
(<year>2012</year>
). <article-title>The effect of whisker movement on radial distance estimation: a case study in comparative robotics</article-title>
. <source>Front. Neurorobot</source>
. <volume>6</volume>
:<issue>12</issue>
. <pub-id pub-id-type="doi">10.3389/fnbot.2012.00012</pub-id>
<pub-id pub-id-type="pmid">23293601</pub-id>
</mixed-citation>
</ref>
<ref id="B21"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Felleman</surname>
<given-names>D. J.</given-names>
</name>
<name><surname>Van Essen</surname>
<given-names>D. C.</given-names>
</name>
</person-group>
(<year>1991</year>
). <article-title>Distributed hierarchical processing in the primate cerebral cortex</article-title>
. <source>Cereb. Cortex</source>
<volume>1</volume>
, <fpage>1</fpage>
–<lpage>47</lpage>
. <pub-id pub-id-type="pmid">1822724</pub-id>
</mixed-citation>
</ref>
<ref id="B22"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ferezou</surname>
<given-names>I.</given-names>
</name>
<name><surname>Haiss</surname>
<given-names>F.</given-names>
</name>
<name><surname>Gentet</surname>
<given-names>L. J.</given-names>
</name>
<name><surname>Aronoff</surname>
<given-names>R.</given-names>
</name>
<name><surname>Weber</surname>
<given-names>B.</given-names>
</name>
<name><surname>Petersen</surname>
<given-names>C. C.</given-names>
</name>
</person-group>
(<year>2007</year>
). <article-title>Spatiotemporal dynamics of cortical sensorimotor integration in behaving mice</article-title>
. <source>Neuron</source>
<volume>56</volume>
, <fpage>907</fpage>
–<lpage>923</lpage>
. <pub-id pub-id-type="doi">10.1016/j.neuron.2007.10.007</pub-id>
<pub-id pub-id-type="pmid">18054865</pub-id>
</mixed-citation>
</ref>
<ref id="B23"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Funahashi</surname>
<given-names>S.</given-names>
</name>
<name><surname>Bruce</surname>
<given-names>C. J.</given-names>
</name>
<name><surname>Goldman-Rakic</surname>
<given-names>P. S.</given-names>
</name>
</person-group>
(<year>1989</year>
). <article-title>Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex</article-title>
. <source>J. Neurophysiol</source>
. <volume>61</volume>
, <fpage>331</fpage>
–<lpage>349</lpage>
. <pub-id pub-id-type="pmid">2918358</pub-id>
</mixed-citation>
</ref>
<ref id="B24"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Gluck</surname>
<given-names>M. A.</given-names>
</name>
<name><surname>Thompson</surname>
<given-names>R. F.</given-names>
</name>
</person-group>
(<year>1987</year>
). <article-title>Modeling the neural substrates of associative learning and memory: a computational approach</article-title>
. <source>Psychol. Rev</source>
. <volume>94</volume>
, <fpage>176</fpage>
–<lpage>191</lpage>
. <pub-id pub-id-type="pmid">3575584</pub-id>
</mixed-citation>
</ref>
<ref id="B25"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Griffith</surname>
<given-names>J. S.</given-names>
</name>
<name><surname>Horn</surname>
<given-names>G.</given-names>
</name>
</person-group>
(<year>1966</year>
). <article-title>An analysis of spontaneous impulse activity of units in the striate cortex of unrestrained cats</article-title>
. <source>J. Physiol</source>
. <volume>186</volume>
, <fpage>516</fpage>
–<lpage>534</lpage>
. <pub-id pub-id-type="pmid">5972156</pub-id>
</mixed-citation>
</ref>
<ref id="B26"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Han</surname>
<given-names>F.</given-names>
</name>
<name><surname>Caporale</surname>
<given-names>N.</given-names>
</name>
<name><surname>Dan</surname>
<given-names>Y.</given-names>
</name>
</person-group>
(<year>2008</year>
). <article-title>Reverberation of recent visual experience in spontaneous cortical waves</article-title>
. <source>Neuron</source>
<volume>60</volume>
, <fpage>321</fpage>
–<lpage>327</lpage>
. <pub-id pub-id-type="doi">10.1016/j.neuron.2008.08.026</pub-id>
<pub-id pub-id-type="pmid">18957223</pub-id>
</mixed-citation>
</ref>
<ref id="B27"><mixed-citation publication-type="book"><person-group person-group-type="author"><name><surname>Houk</surname>
<given-names>J. C.</given-names>
</name>
<name><surname>Adams</surname>
<given-names>J. L.</given-names>
</name>
<name><surname>Barto</surname>
<given-names>A. G.</given-names>
</name>
</person-group>
(<year>1995</year>
). <article-title>A model of how the basal ganglia generate and use neural signals that predict reinforcement</article-title>
, in <source>Models of Information Processing in the Basal Ganglia</source>
, eds <person-group person-group-type="editor"><name><surname>Houk</surname>
<given-names>J. C.</given-names>
</name>
<name><surname>Davis</surname>
<given-names>J. L.</given-names>
</name>
<name><surname>Beiser</surname>
<given-names>D. G.</given-names>
</name>
</person-group>
(<publisher-loc>Cambridge, MA</publisher-loc>
: <publisher-name>MIT Press</publisher-name>
), <fpage>249</fpage>
–<lpage>270</lpage>
.</mixed-citation>
</ref>
<ref id="B28"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Humphries</surname>
<given-names>M. D.</given-names>
</name>
<name><surname>Lepora</surname>
<given-names>N.</given-names>
</name>
<name><surname>Wood</surname>
<given-names>R.</given-names>
</name>
<name><surname>Gurney</surname>
<given-names>K.</given-names>
</name>
</person-group>
(<year>2009</year>
). <article-title>Capturing dopaminergic modulation and bimodal membrane behaviour of striatal medium spiny neurons in accurate, reduced models</article-title>
. <source>Front. Comput. Neurosci</source>
. <volume>3</volume>
:<fpage>26</fpage>
. <pub-id pub-id-type="doi">10.3389/neuro.10.026.2009</pub-id>
<pub-id pub-id-type="pmid">20011223</pub-id>
</mixed-citation>
</ref>
<ref id="B29"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Izhikevich</surname>
<given-names>E. M.</given-names>
</name>
</person-group>
(<year>2003</year>
). <article-title>Simple model of spiking neurons</article-title>
. <source>IEEE Trans. Neural Netw</source>
. <volume>14</volume>
, <fpage>1569</fpage>
–<lpage>1572</lpage>
. <pub-id pub-id-type="doi">10.1109/TNN.2003.820440</pub-id>
<pub-id pub-id-type="pmid">18244602</pub-id>
</mixed-citation>
</ref>
<ref id="B30"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Izhikevich</surname>
<given-names>E. M.</given-names>
</name>
</person-group>
(<year>2006</year>
). <article-title>Polychronization: computation with spikes</article-title>
. <source>Neural Comput</source>
. <volume>18</volume>
, <fpage>245</fpage>
–<lpage>282</lpage>
. <pub-id pub-id-type="doi">10.1162/089976606775093882</pub-id>
<pub-id pub-id-type="pmid">16378515</pub-id>
</mixed-citation>
</ref>
<ref id="B31"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Izhikevich</surname>
<given-names>E. M.</given-names>
</name>
</person-group>
(<year>2007</year>
). <article-title>Solving the distal reward problem through linkage of STDP and dopamine signaling</article-title>
. <source>Cereb. Cortex</source>
<volume>17</volume>
, <fpage>2443</fpage>
–<lpage>2452</lpage>
. <pub-id pub-id-type="doi">10.1093/cercor/bhl152</pub-id>
<pub-id pub-id-type="pmid">17220510</pub-id>
</mixed-citation>
</ref>
<ref id="B32"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Izhikevich</surname>
<given-names>E. M.</given-names>
</name>
<name><surname>Edelman</surname>
<given-names>G. M.</given-names>
</name>
</person-group>
(<year>2008</year>
). <article-title>Large-scale model of mammalian thalamocortical systems</article-title>
. <source>Proc. Natl. Acad. Sci. U.S.A</source>
. <volume>105</volume>
, <fpage>3593</fpage>
–<lpage>3598</lpage>
. <pub-id pub-id-type="doi">10.1073/pnas.0712231105</pub-id>
<pub-id pub-id-type="pmid">18292226</pub-id>
</mixed-citation>
</ref>
<ref id="B33"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Khamassi</surname>
<given-names>M.</given-names>
</name>
<name><surname>Humphries</surname>
<given-names>M. D.</given-names>
</name>
</person-group>
(<year>2012</year>
). <article-title>Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies</article-title>
. <source>Front. Behav. Neurosci</source>
. <volume>6</volume>
:<issue>79</issue>
. <pub-id pub-id-type="doi">10.3389/fnbeh.2012.00079</pub-id>
<pub-id pub-id-type="pmid">23205006</pub-id>
</mixed-citation>
</ref>
<ref id="B34"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Khamassi</surname>
<given-names>M.</given-names>
</name>
<name><surname>Lallee</surname>
<given-names>S.</given-names>
</name>
<name><surname>Enel</surname>
<given-names>P.</given-names>
</name>
<name><surname>Procyk</surname>
<given-names>E.</given-names>
</name>
<name><surname>Dominey</surname>
<given-names>P. F.</given-names>
</name>
</person-group>
(<year>2011</year>
). <article-title>Robot cognitive control with a neurophysiologically inspired reinforcement learning model</article-title>
. <source>Front. Neurorobot</source>
. <volume>5</volume>
:<issue>1</issue>
. <pub-id pub-id-type="doi">10.3389/fnbot.2011.00001</pub-id>
<pub-id pub-id-type="pmid">21808619</pub-id>
</mixed-citation>
</ref>
<ref id="B35"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Koch</surname>
<given-names>K. W.</given-names>
</name>
<name><surname>Fuster</surname>
<given-names>J. M.</given-names>
</name>
</person-group>
(<year>1989</year>
). <article-title>Unit activity in monkey parietal cortex related to haptic perception and temporary memory</article-title>
. <source>Exp. Brain Res</source>
. <volume>76</volume>
, <fpage>292</fpage>
–<lpage>306</lpage>
. <pub-id pub-id-type="pmid">2767186</pub-id>
</mixed-citation>
</ref>
<ref id="B36"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Krichmar</surname>
<given-names>J. L.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>A neurorobotic platform to test the influence of neuromodulatory signaling on anxious and curious behavior</article-title>
. <source>Front. Neurorobot</source>
. <volume>7</volume>
:<issue>1</issue>
. <pub-id pub-id-type="doi">10.3389/fnbot.2013.00001</pub-id>
<pub-id pub-id-type="pmid">23386829</pub-id>
</mixed-citation>
</ref>
<ref id="B37"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Krichmar</surname>
<given-names>J. L.</given-names>
</name>
<name><surname>Rohrbein</surname>
<given-names>F.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>Value and reward based learning in neurorobots</article-title>
. <source>Front. Neurorobot</source>
. <volume>7</volume>
:<issue>13</issue>
. <pub-id pub-id-type="doi">10.3389/fnbot.2013.00013</pub-id>
<pub-id pub-id-type="pmid">24062683</pub-id>
</mixed-citation>
</ref>
<ref id="B38"><mixed-citation publication-type="book"><person-group person-group-type="author"><name><surname>Li</surname>
<given-names>Q.</given-names>
</name>
<name><surname>Elbrechter</surname>
<given-names>C.</given-names>
</name>
<name><surname>Haschke</surname>
<given-names>R.</given-names>
</name>
<name><surname>Ritter</surname>
<given-names>H.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>Integrating vision, haptics and proprioception into a feedback controller for in-hand manipulation of unknown objects</article-title>
, in <source>Paper Presented at the Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on</source>
(<publisher-loc>Tokyo</publisher-loc>
).</mixed-citation>
</ref>
<ref id="B39"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ljungberg</surname>
<given-names>T.</given-names>
</name>
<name><surname>Apicella</surname>
<given-names>P.</given-names>
</name>
<name><surname>Schultz</surname>
<given-names>W.</given-names>
</name>
</person-group>
(<year>1991</year>
). <article-title>Responses of monkey midbrain dopamine neurons during delayed alternation performance</article-title>
. <source>Brain Res</source>
. <volume>567</volume>
, <fpage>337</fpage>
–<lpage>341</lpage>
. <pub-id pub-id-type="pmid">1817739</pub-id>
</mixed-citation>
</ref>
<ref id="B40"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ljungberg</surname>
<given-names>T.</given-names>
</name>
<name><surname>Apicella</surname>
<given-names>P.</given-names>
</name>
<name><surname>Schultz</surname>
<given-names>W.</given-names>
</name>
</person-group>
(<year>1992</year>
). <article-title>Responses of monkey dopamine neurons during learning of behavioral reactions</article-title>
. <source>J. Neurophysiol</source>
. <volume>67</volume>
, <fpage>145</fpage>
–<lpage>163</lpage>
. <pub-id pub-id-type="pmid">1552316</pub-id>
</mixed-citation>
</ref>
<ref id="B41"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Lubenov</surname>
<given-names>E. V.</given-names>
</name>
<name><surname>Siapas</surname>
<given-names>A. G.</given-names>
</name>
</person-group>
(<year>2009</year>
). <article-title>Hippocampal theta oscillations are travelling waves</article-title>
. <source>Nature</source>
<volume>459</volume>
, <fpage>534</fpage>
–<lpage>539</lpage>
. <pub-id pub-id-type="doi">10.1038/nature08010</pub-id>
<pub-id pub-id-type="pmid">19489117</pub-id>
</mixed-citation>
</ref>
<ref id="B42"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Maheshwari</surname>
<given-names>V.</given-names>
</name>
<name><surname>Saraf</surname>
<given-names>R.</given-names>
</name>
</person-group>
(<year>2008</year>
). <article-title>Tactile devices to sense touch on a par with a human finger</article-title>
. <source>Ang. Chem. Intern. Edn</source>
. <volume>47</volume>
, <fpage>7808</fpage>
–<lpage>7826</lpage>
. <pub-id pub-id-type="doi">10.1002/anie.200703693</pub-id>
<pub-id pub-id-type="pmid">18816579</pub-id>
</mixed-citation>
</ref>
<ref id="B43"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Markram</surname>
<given-names>H.</given-names>
</name>
<name><surname>Gerstner</surname>
<given-names>W.</given-names>
</name>
<name><surname>Sjostrom</surname>
<given-names>P. J.</given-names>
</name>
</person-group>
(<year>2011</year>
). <article-title>A history of spike-timing-dependent plasticity</article-title>
. <source>Front. Synaptic Neurosci</source>
. <volume>3</volume>
:<issue>4</issue>
. <pub-id pub-id-type="doi">10.3389/fnsyn.2011.00004</pub-id>
<pub-id pub-id-type="pmid">22007168</pub-id>
</mixed-citation>
</ref>
<ref id="B44"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Miyazaki</surname>
<given-names>K. W.</given-names>
</name>
<name><surname>Miyazaki</surname>
<given-names>K.</given-names>
</name>
<name><surname>Doya</surname>
<given-names>K.</given-names>
</name>
</person-group>
(<year>2012</year>
). <article-title>Activation of dorsal raphe serotonin neurons is necessary for waiting for delayed rewards</article-title>
. <source>J. Neurosci</source>
. <volume>32</volume>
, <fpage>10451</fpage>
–<lpage>10457</lpage>
. <pub-id pub-id-type="doi">10.1523/JNEUROSCI.0915-12.2012</pub-id>
<pub-id pub-id-type="pmid">22855794</pub-id>
</mixed-citation>
</ref>
<ref id="B45"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Montague</surname>
<given-names>P. R.</given-names>
</name>
<name><surname>Dayan</surname>
<given-names>P.</given-names>
</name>
<name><surname>Sejnowski</surname>
<given-names>T. J.</given-names>
</name>
</person-group>
(<year>1996</year>
). <article-title>A framework for mesencephalic dopamine systems based on predictive Hebbian learning</article-title>
. <source>J. Neurosci</source>
. <volume>16</volume>
, <fpage>1936</fpage>
–<lpage>1947</lpage>
. <pub-id pub-id-type="pmid">8774460</pub-id>
</mixed-citation>
</ref>
<ref id="B46"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Morrison</surname>
<given-names>I.</given-names>
</name>
<name><surname>Bjornsdotter</surname>
<given-names>M.</given-names>
</name>
<name><surname>Olausson</surname>
<given-names>H.</given-names>
</name>
</person-group>
(<year>2011a</year>
). <article-title>Vicarious responses to social touch in posterior insular cortex are tuned to pleasant caressing speeds</article-title>
. <source>J. Neurosci</source>
. <volume>31</volume>
, <fpage>9554</fpage>
–<lpage>9562</lpage>
. <pub-id pub-id-type="doi">10.1523/JNEUROSCI.0397-11.2011</pub-id>
<pub-id pub-id-type="pmid">21715620</pub-id>
</mixed-citation>
</ref>
<ref id="B47"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Morrison</surname>
<given-names>I.</given-names>
</name>
<name><surname>Loken</surname>
<given-names>L. S.</given-names>
</name>
<name><surname>Minde</surname>
<given-names>J.</given-names>
</name>
<name><surname>Wessberg</surname>
<given-names>J.</given-names>
</name>
<name><surname>Perini</surname>
<given-names>I.</given-names>
</name>
<name><surname>Nennesmo</surname>
<given-names>I.</given-names>
</name>
<etal></etal>
</person-group>
. (<year>2011b</year>
). Reduced C-afferent fibre density affects perceived pleasantness and empathy for touch. <italic>Brain</italic>
<volume>134(Pt 4)</volume>
, <fpage>1116</fpage>
–<lpage>1126</lpage>
. <pub-id pub-id-type="doi">10.1093/brain/awr011</pub-id>
<pub-id pub-id-type="pmid">21378097</pub-id>
</mixed-citation>
</ref>
<ref id="B48"><mixed-citation publication-type="book"><person-group person-group-type="author"><name><surname>N'Guyen</surname>
<given-names>S.</given-names>
</name>
<name><surname>Pirim</surname>
<given-names>P.</given-names>
</name>
<name><surname>Meyer</surname>
<given-names>J.-A.</given-names>
</name>
</person-group>
(<year>2010</year>
). <article-title>Tactile texture discrimination in the robot-rat psikharpax</article-title>
, in <source>Paper Presented at the BIOSIGNALS</source>
(<publisher-loc>Valencia</publisher-loc>
).</mixed-citation>
</ref>
<ref id="B49"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Nageswaran</surname>
<given-names>J. M.</given-names>
</name>
<name><surname>Dutt</surname>
<given-names>N.</given-names>
</name>
<name><surname>Krichmar</surname>
<given-names>J. L.</given-names>
</name>
<name><surname>Nicolau</surname>
<given-names>A.</given-names>
</name>
<name><surname>Veidenbaum</surname>
<given-names>A. V.</given-names>
</name>
</person-group>
(<year>2009</year>
). <article-title>A configurable simulation environment for the efficient simulation of large-scale spiking neural networks on graphics processors</article-title>
. <source>Neural Netw</source>
. <volume>22</volume>
, <fpage>791</fpage>
–<lpage>800</lpage>
. <pub-id pub-id-type="doi">10.1016/j.neunet.2009.06.028</pub-id>
<pub-id pub-id-type="pmid">19615853</pub-id>
</mixed-citation>
</ref>
<ref id="B50"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Nicola</surname>
<given-names>S. M.</given-names>
</name>
<name><surname>Surmeier</surname>
<given-names>J.</given-names>
</name>
<name><surname>Malenka</surname>
<given-names>R. C.</given-names>
</name>
</person-group>
(<year>2000</year>
). <article-title>Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens</article-title>
. <source>Annu. Rev. Neurosci</source>
. <volume>23</volume>
, <fpage>185</fpage>
–<lpage>215</lpage>
. <pub-id pub-id-type="doi">10.1146/annurev.neuro.23.1.185</pub-id>
<pub-id pub-id-type="pmid">10845063</pub-id>
</mixed-citation>
</ref>
<ref id="B51"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Olausson</surname>
<given-names>H.</given-names>
</name>
<name><surname>Wessberg</surname>
<given-names>J.</given-names>
</name>
<name><surname>Morrison</surname>
<given-names>I.</given-names>
</name>
<name><surname>McGlone</surname>
<given-names>F.</given-names>
</name>
<name><surname>Vallbo</surname>
<given-names>A.</given-names>
</name>
</person-group>
(<year>2010</year>
). <article-title>The neurophysiology of unmyelinated tactile afferents</article-title>
. <source>Neurosci. Biobehav. Rev</source>
. <volume>34</volume>
, <fpage>185</fpage>
–<lpage>191</lpage>
. <pub-id pub-id-type="doi">10.1016/j.neubiorev.2008.09.011</pub-id>
<pub-id pub-id-type="pmid">18952123</pub-id>
</mixed-citation>
</ref>
<ref id="B52"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Palmer</surname>
<given-names>J. H.</given-names>
</name>
<name><surname>Gong</surname>
<given-names>P.</given-names>
</name>
</person-group>
(<year>2014</year>
). <article-title>Associative learning of classical conditioning as an emergent property of spatially extended spiking neural circuits with synaptic plasticity</article-title>
. <source>Front. Comput. Neurosci</source>
. <volume>8</volume>
:<issue>79</issue>
. <pub-id pub-id-type="doi">10.3389/fncom.2014.00079</pub-id>
<pub-id pub-id-type="pmid">25120462</pub-id>
</mixed-citation>
</ref>
<ref id="B53"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Pan</surname>
<given-names>W. X.</given-names>
</name>
<name><surname>Schmidt</surname>
<given-names>R.</given-names>
</name>
<name><surname>Wickens</surname>
<given-names>J. R.</given-names>
</name>
<name><surname>Hyland</surname>
<given-names>B. I.</given-names>
</name>
</person-group>
(<year>2005</year>
). <article-title>Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network</article-title>
. <source>J. Neurosci</source>
. <volume>25</volume>
, <fpage>6235</fpage>
–<lpage>6242</lpage>
. <pub-id pub-id-type="doi">10.1523/JNEUROSCI.1478-05.2005</pub-id>
<pub-id pub-id-type="pmid">15987953</pub-id>
</mixed-citation>
</ref>
<ref id="B54"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Pearson</surname>
<given-names>M. J.</given-names>
</name>
<name><surname>Mitchinson</surname>
<given-names>B.</given-names>
</name>
<name><surname>Sullivan</surname>
<given-names>J. C.</given-names>
</name>
<name><surname>Pipe</surname>
<given-names>A. G.</given-names>
</name>
<name><surname>Prescott</surname>
<given-names>T. J.</given-names>
</name>
</person-group>
(<year>2011</year>
). <article-title>Biomimetic vibrissal sensing for robots</article-title>
. <source>Philos. Trans. R. Soc. Lond. B Biol. Sci</source>
. <volume>366</volume>
, <fpage>3085</fpage>
–<lpage>3096</lpage>
. <pub-id pub-id-type="doi">10.1098/rstb.2011.0164</pub-id>
<pub-id pub-id-type="pmid">21969690</pub-id>
</mixed-citation>
</ref>
<ref id="B55"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rescorla</surname>
<given-names>R. A.</given-names>
</name>
</person-group>
(<year>1988</year>
). <article-title>Behavioral studies of Pavlovian conditioning</article-title>
. <source>Annu. Rev. Neurosci</source>
. <volume>11</volume>
, <fpage>329</fpage>
–<lpage>352</lpage>
. <pub-id pub-id-type="doi">10.1146/annurev.ne.11.030188.001553</pub-id>
<pub-id pub-id-type="pmid">3284445</pub-id>
</mixed-citation>
</ref>
<ref id="B56"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Richert</surname>
<given-names>M.</given-names>
</name>
<name><surname>Nageswaran</surname>
<given-names>J. M.</given-names>
</name>
<name><surname>Dutt</surname>
<given-names>N.</given-names>
</name>
<name><surname>Krichmar</surname>
<given-names>J. L.</given-names>
</name>
</person-group>
(<year>2011</year>
). <article-title>An efficient simulation environment for modeling large-scale cortical processing</article-title>
. <source>Front. Neuroinform</source>
. <volume>5</volume>
:<issue>19</issue>
. <pub-id pub-id-type="doi">10.3389/fninf.2011.00019</pub-id>
<pub-id pub-id-type="pmid">22007166</pub-id>
</mixed-citation>
</ref>
<ref id="B57"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rubino</surname>
<given-names>D.</given-names>
</name>
<name><surname>Robbins</surname>
<given-names>K. A.</given-names>
</name>
<name><surname>Hatsopoulos</surname>
<given-names>N. G.</given-names>
</name>
</person-group>
(<year>2006</year>
). <article-title>Propagating waves mediate information transfer in the motor cortex</article-title>
. <source>Nat. Neurosci</source>
. <volume>9</volume>
, <fpage>1549</fpage>
–<lpage>1557</lpage>
. <pub-id pub-id-type="doi">10.1038/nn1802</pub-id>
<pub-id pub-id-type="pmid">17115042</pub-id>
</mixed-citation>
</ref>
<ref id="B58"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sato</surname>
<given-names>T. K.</given-names>
</name>
<name><surname>Nauhaus</surname>
<given-names>I.</given-names>
</name>
<name><surname>Carandini</surname>
<given-names>M.</given-names>
</name>
</person-group>
(<year>2012</year>
). <article-title>Traveling waves in visual cortex</article-title>
. <source>Neuron</source>
<volume>75</volume>
, <fpage>218</fpage>
–<lpage>229</lpage>
. <pub-id pub-id-type="doi">10.1016/j.neuron.2012.06.029</pub-id>
<pub-id pub-id-type="pmid">22841308</pub-id>
</mixed-citation>
</ref>
<ref id="B59"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Scassellati</surname>
<given-names>B.</given-names>
</name>
<name><surname>Admoni</surname>
<given-names>H.</given-names>
</name>
<name><surname>Mataric</surname>
<given-names>M.</given-names>
</name>
</person-group>
(<year>2012</year>
). <article-title>Robots for use in autism research</article-title>
. <source>Annu. Rev. Biomed. Eng</source>
. <volume>14</volume>
, <fpage>275</fpage>
–<lpage>294</lpage>
. <pub-id pub-id-type="doi">10.1146/annurev-bioeng-071811-150036</pub-id>
<pub-id pub-id-type="pmid">22577778</pub-id>
</mixed-citation>
</ref>
<ref id="B60"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Schroeder</surname>
<given-names>C. L.</given-names>
</name>
<name><surname>Hartmann</surname>
<given-names>M. J.</given-names>
</name>
</person-group>
(<year>2012</year>
). <article-title>Sensory prediction on a whiskered robot: a tactile analogy to “optical flow”</article-title>
. <source>Front. Neurorobot</source>
. <volume>6</volume>
:<issue>9</issue>
. <pub-id pub-id-type="doi">10.3389/fnbot.2012.00009</pub-id>
<pub-id pub-id-type="pmid">23097641</pub-id>
</mixed-citation>
</ref>
<ref id="B61"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Schultz</surname>
<given-names>W.</given-names>
</name>
</person-group>
(<year>1998</year>
). <article-title>Predictive reward signal of dopamine neurons</article-title>
. <source>J. Neurophysiol</source>
. <volume>80</volume>
, <fpage>1</fpage>
–<lpage>27</lpage>
. <pub-id pub-id-type="pmid">9658025</pub-id>
</mixed-citation>
</ref>
<ref id="B62"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Schultz</surname>
<given-names>W.</given-names>
</name>
</person-group>
(<year>2006</year>
). <article-title>Behavioral theories and the neurophysiology of reward</article-title>
. <source>Annu. Rev. Psychol</source>
. <volume>57</volume>
, <fpage>87</fpage>
–<lpage>115</lpage>
. <pub-id pub-id-type="doi">10.1146/annurev.psych.56.091103.070229</pub-id>
<pub-id pub-id-type="pmid">16318590</pub-id>
</mixed-citation>
</ref>
<ref id="B63"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Schultz</surname>
<given-names>W.</given-names>
</name>
<name><surname>Dayan</surname>
<given-names>P.</given-names>
</name>
<name><surname>Montague</surname>
<given-names>P. R.</given-names>
</name>
</person-group>
(<year>1997</year>
). <article-title>A neural substrate of prediction and reward</article-title>
. <source>Science</source>
<volume>275</volume>
, <fpage>1593</fpage>
–<lpage>1599</lpage>
. <pub-id pub-id-type="pmid">9054347</pub-id>
</mixed-citation>
</ref>
<ref id="B64"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Seth</surname>
<given-names>A. K.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>Interoceptive inference, emotion, and the embodied self</article-title>
. <source>Trends Cogn. Sci</source>
. <volume>17</volume>
, <fpage>565</fpage>
–<lpage>573</lpage>
. <pub-id pub-id-type="doi">10.1016/j.tics.2013.09.007</pub-id>
<pub-id pub-id-type="pmid">24126130</pub-id>
</mixed-citation>
</ref>
<ref id="B65"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sewards</surname>
<given-names>T. V.</given-names>
</name>
<name><surname>Sewards</surname>
<given-names>M.</given-names>
</name>
</person-group>
(<year>2002</year>
). <article-title>Separate, parallel sensory and hedonic pathways in the mammalian somatosensory system</article-title>
. <source>Brain Res. Bull</source>
. <volume>58</volume>
, <fpage>243</fpage>
–<lpage>260</lpage>
. <pub-id pub-id-type="doi">10.1016/S0361-9230(02)00783-9</pub-id>
<pub-id pub-id-type="pmid">12128150</pub-id>
</mixed-citation>
</ref>
<ref id="B66"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Siegel</surname>
<given-names>J. Z.</given-names>
</name>
<name><surname>Crockett</surname>
<given-names>M. J.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>How serotonin shapes moral judgment and behavior</article-title>
. <source>Ann. N.Y. Acad. Sci</source>
. <volume>1299</volume>
, <fpage>42</fpage>
–<lpage>51</lpage>
. <pub-id pub-id-type="doi">10.1111/nyas.12229</pub-id>
<pub-id pub-id-type="pmid">25627116</pub-id>
</mixed-citation>
</ref>
<ref id="B67"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Silver</surname>
<given-names>R. A.</given-names>
</name>
</person-group>
(<year>2010</year>
). <article-title>Neuronal arithmetic</article-title>
. <source>Nat. Rev. Neurosci</source>
. <volume>11</volume>
, <fpage>474</fpage>
–<lpage>489</lpage>
. <pub-id pub-id-type="doi">10.1038/nrn2864</pub-id>
<pub-id pub-id-type="pmid">20531421</pub-id>
</mixed-citation>
</ref>
<ref id="B68"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Singer</surname>
<given-names>T.</given-names>
</name>
<name><surname>Critchley</surname>
<given-names>H. D.</given-names>
</name>
<name><surname>Preuschoff</surname>
<given-names>K.</given-names>
</name>
</person-group>
(<year>2009</year>
). <article-title>A common role of insula in feelings, empathy and uncertainty</article-title>
. <source>Trends Cogn. Sci</source>
. <volume>13</volume>
, <fpage>334</fpage>
–<lpage>340</lpage>
. <pub-id pub-id-type="doi">10.1016/j.tics.2009.05.001</pub-id>
<pub-id pub-id-type="pmid">19643659</pub-id>
</mixed-citation>
</ref>
<ref id="B69"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Soltoggio</surname>
<given-names>A.</given-names>
</name>
<name><surname>Lemme</surname>
<given-names>A.</given-names>
</name>
<name><surname>Reinhart</surname>
<given-names>F.</given-names>
</name>
<name><surname>Steil</surname>
<given-names>J. J.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>Rare neural correlations implement robotic conditioning with delayed rewards and disturbances</article-title>
. <source>Front. Neurorobot</source>
. <volume>7</volume>
:<issue>6</issue>
. <pub-id pub-id-type="doi">10.3389/fnbot.2013.00006</pub-id>
<pub-id pub-id-type="pmid">23565092</pub-id>
</mixed-citation>
</ref>
<ref id="B70"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Soltoggio</surname>
<given-names>A.</given-names>
</name>
<name><surname>Steil</surname>
<given-names>J. J.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>Solving the distal reward problem with rare correlations</article-title>
. <source>Neural Comput</source>
. <volume>25</volume>
, <fpage>940</fpage>
–<lpage>978</lpage>
. <pub-id pub-id-type="doi">10.1162/NECO_a_00419</pub-id>
<pub-id pub-id-type="pmid">23339615</pub-id>
</mixed-citation>
</ref>
<ref id="B71"><mixed-citation publication-type="book"><person-group person-group-type="author"><name><surname>Spigler</surname>
<given-names>G.</given-names>
</name>
<name><surname>Oddo</surname>
<given-names>C. M.</given-names>
</name>
<name><surname>Carrozza</surname>
<given-names>M. C.</given-names>
</name>
</person-group>
(<year>2012</year>
). <article-title>Soft-neuromorphic artificial touch for applications in neuro-robotics</article-title>
, in <source>Paper Presented at the Biomedical Robotics and Biomechatronics (BioRob), 2012 4th IEEE RAS & EMBS International Conference on</source>
(<publisher-loc>Rome</publisher-loc>
).</mixed-citation>
</ref>
<ref id="B72"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Srinivasa</surname>
<given-names>N.</given-names>
</name>
<name><surname>Jiang</surname>
<given-names>Q.</given-names>
</name>
</person-group>
(<year>2013</year>
). <article-title>Stable learning of functional maps in self-organizing spiking neural networks with continuous synaptic plasticity</article-title>
. <source>Front. Comput. Neurosci</source>
. <volume>7</volume>
:<issue>10</issue>
. <pub-id pub-id-type="doi">10.3389/fncom.2013.00010</pub-id>
<pub-id pub-id-type="pmid">23450808</pub-id>
</mixed-citation>
</ref>
<ref id="B73"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Szatmary</surname>
<given-names>B.</given-names>
</name>
<name><surname>Izhikevich</surname>
<given-names>E. M.</given-names>
</name>
</person-group>
(<year>2010</year>
). <article-title>Spike-timing theory of working memory</article-title>
. <source>PLoS Comput. Biol</source>
. <volume>6</volume>
:<fpage>879</fpage>
. <pub-id pub-id-type="doi">10.1371/journal.pcbi.1000879</pub-id>
<pub-id pub-id-type="pmid">20808877</pub-id>
</mixed-citation>
</ref>
<ref id="B74"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Tan</surname>
<given-names>C. O.</given-names>
</name>
<name><surname>Bullock</surname>
<given-names>D.</given-names>
</name>
</person-group>
(<year>2008</year>
). <article-title>A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward</article-title>
. <source>J. Neurosci</source>
. <volume>28</volume>
, <fpage>10062</fpage>
–<lpage>10074</lpage>
. <pub-id pub-id-type="doi">10.1523/JNEUROSCI.0259-08.2008</pub-id>
<pub-id pub-id-type="pmid">18829964</pub-id>
</mixed-citation>
</ref>
<ref id="B75"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Tops</surname>
<given-names>M.</given-names>
</name>
<name><surname>Russo</surname>
<given-names>S.</given-names>
</name>
<name><surname>Boksem</surname>
<given-names>M. A.</given-names>
</name>
<name><surname>Tucker</surname>
<given-names>D. M.</given-names>
</name>
</person-group>
(<year>2009</year>
). <article-title>Serotonin: modulator of a drive to withdraw</article-title>
. <source>Brain Cogn</source>
. <volume>71</volume>
, <fpage>427</fpage>
–<lpage>436</lpage>
. <pub-id pub-id-type="doi">10.1016/j.bandc.2009.03.009</pub-id>
<pub-id pub-id-type="pmid">19423206</pub-id>
</mixed-citation>
</ref>
<ref id="B76"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Williams</surname>
<given-names>G. V.</given-names>
</name>
<name><surname>Castner</surname>
<given-names>S. A.</given-names>
</name>
</person-group>
(<year>2006</year>
). <article-title>Under the curve: critical issues for elucidating D1 receptor function in working memory</article-title>
. <source>Neuroscience</source>
<volume>139</volume>
, <fpage>263</fpage>
–<lpage>276</lpage>
. <pub-id pub-id-type="doi">10.1016/j.neuroscience.2005.09.028</pub-id>
<pub-id pub-id-type="pmid">16310964</pub-id>
</mixed-citation>
</ref>
<ref id="B77"><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Wu</surname>
<given-names>J. Y.</given-names>
</name>
<name><surname>Xiaoying</surname>
<given-names>H.</given-names>
</name>
<name><surname>Chuan</surname>
<given-names>Z.</given-names>
</name>
</person-group>
(<year>2008</year>
). <article-title>Propagating waves of activity in the neocortex: what they are, what they do</article-title>
. <source>Neuroscientist</source>
<volume>14</volume>
, <fpage>487</fpage>
–<lpage>502</lpage>
. <pub-id pub-id-type="doi">10.1177/1073858408317066</pub-id>
<pub-id pub-id-type="pmid">18997124</pub-id>
</mixed-citation>
</ref>
</ref-list>
</back>
</pmc>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/HapticV1/Data/Pmc/Curation
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000187 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Curation/biblio.hfd -nk 000187 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Ticri/CIDE |area= HapticV1 |flux= Pmc |étape= Curation |type= RBID |clé= PMC:4510776 |texte= Learning touch preferences with a tactile robot using dopamine modulated STDP in a model of insular cortex }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Curation/RBID.i -Sk "pubmed:26257639" \ | HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Curation/biblio.hfd \ | NlmPubMed2Wicri -a HapticV1
![]() | This area was generated with Dilib version V0.6.23. | ![]() |