Credit Assignment during Movement Reinforcement Learning
Identifieur interne : 002522 ( Ncbi/Merge ); précédent : 002521; suivant : 002523Credit Assignment during Movement Reinforcement Learning
Auteurs : Gregory Dam [États-Unis] ; Konrad Kording [États-Unis] ; Kunlin Wei [République populaire de Chine]Source :
- PLoS ONE [ 1932-6203 ] ; 2013.
Abstract
We often need to learn how to move based on a single performance measure that reflects the overall success of our movements. However, movements have many properties, such as their trajectories, speeds and timing of end-points, thus the brain needs to decide which properties of movements should be improved; it needs to solve the credit assignment problem. Currently, little is known about how humans solve credit assignment problems in the context of reinforcement learning. Here we tested how human participants solve such problems during a trajectory-learning task. Without an explicitly-defined target movement, participants made hand reaches and received monetary rewards as feedback on a trial-by-trial basis. The curvature and direction of the attempted reach trajectories determined the monetary rewards received in a manner that can be manipulated experimentally. Based on the history of action-reward pairs, participants quickly solved the credit assignment problem and learned the implicit payoff function. A Bayesian credit-assignment model with built-in forgetting accurately predicts their trial-by-trial learning.
Url:
DOI: 10.1371/journal.pone.0055352
PubMed: 23408972
PubMed Central: 3568147
Links toward previous steps (curation, corpus...)
- to stream Pmc, to step Corpus: 002254
- to stream Pmc, to step Curation: 002254
- to stream Pmc, to step Checkpoint: 001339
Links to Exploration step
PMC:3568147Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Credit Assignment during Movement Reinforcement Learning</title>
<author><name sortKey="Dam, Gregory" sort="Dam, Gregory" uniqKey="Dam G" first="Gregory" last="Dam">Gregory Dam</name>
<affiliation wicri:level="2"><nlm:aff id="aff1"><addr-line>Department of Behavioral Sciences, University of Rio Grande, Rio Grande, Ohio, United States of America</addr-line>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>Department of Behavioral Sciences, University of Rio Grande, Rio Grande, Ohio</wicri:regionArea>
<placeName><region type="state">Ohio</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Kording, Konrad" sort="Kording, Konrad" uniqKey="Kording K" first="Konrad" last="Kording">Konrad Kording</name>
<affiliation wicri:level="2"><nlm:aff id="aff2"><addr-line>Department of Physical Medicine and Rehabilitation, Rehabilitation Institute of Chicago, Northwestern University, Chicago, Illinois, United States of America</addr-line>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>Department of Physical Medicine and Rehabilitation, Rehabilitation Institute of Chicago, Northwestern University, Chicago, Illinois</wicri:regionArea>
<placeName><region type="state">Illinois</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Wei, Kunlin" sort="Wei, Kunlin" uniqKey="Wei K" first="Kunlin" last="Wei">Kunlin Wei</name>
<affiliation wicri:level="4"><nlm:aff id="aff3"><addr-line>Department of Psychology, Key Laboratory of Machine Perception (Ministry of Education), Beijing Engineering Research Center of Intelligent Rehabilitation Engineering, Peking University, Beijing, China</addr-line>
</nlm:aff>
<country xml:lang="fr">République populaire de Chine</country>
<wicri:regionArea>Department of Psychology, Key Laboratory of Machine Perception (Ministry of Education), Beijing Engineering Research Center of Intelligent Rehabilitation Engineering, Peking University, Beijing</wicri:regionArea>
<placeName><settlement type="city">Pékin</settlement>
</placeName>
<orgName type="university">Université de Pékin</orgName>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PMC</idno>
<idno type="pmid">23408972</idno>
<idno type="pmc">3568147</idno>
<idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3568147</idno>
<idno type="RBID">PMC:3568147</idno>
<idno type="doi">10.1371/journal.pone.0055352</idno>
<date when="2013">2013</date>
<idno type="wicri:Area/Pmc/Corpus">002254</idno>
<idno type="wicri:Area/Pmc/Curation">002254</idno>
<idno type="wicri:Area/Pmc/Checkpoint">001339</idno>
<idno type="wicri:Area/Ncbi/Merge">002522</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Credit Assignment during Movement Reinforcement Learning</title>
<author><name sortKey="Dam, Gregory" sort="Dam, Gregory" uniqKey="Dam G" first="Gregory" last="Dam">Gregory Dam</name>
<affiliation wicri:level="2"><nlm:aff id="aff1"><addr-line>Department of Behavioral Sciences, University of Rio Grande, Rio Grande, Ohio, United States of America</addr-line>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>Department of Behavioral Sciences, University of Rio Grande, Rio Grande, Ohio</wicri:regionArea>
<placeName><region type="state">Ohio</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Kording, Konrad" sort="Kording, Konrad" uniqKey="Kording K" first="Konrad" last="Kording">Konrad Kording</name>
<affiliation wicri:level="2"><nlm:aff id="aff2"><addr-line>Department of Physical Medicine and Rehabilitation, Rehabilitation Institute of Chicago, Northwestern University, Chicago, Illinois, United States of America</addr-line>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>Department of Physical Medicine and Rehabilitation, Rehabilitation Institute of Chicago, Northwestern University, Chicago, Illinois</wicri:regionArea>
<placeName><region type="state">Illinois</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Wei, Kunlin" sort="Wei, Kunlin" uniqKey="Wei K" first="Kunlin" last="Wei">Kunlin Wei</name>
<affiliation wicri:level="4"><nlm:aff id="aff3"><addr-line>Department of Psychology, Key Laboratory of Machine Perception (Ministry of Education), Beijing Engineering Research Center of Intelligent Rehabilitation Engineering, Peking University, Beijing, China</addr-line>
</nlm:aff>
<country xml:lang="fr">République populaire de Chine</country>
<wicri:regionArea>Department of Psychology, Key Laboratory of Machine Perception (Ministry of Education), Beijing Engineering Research Center of Intelligent Rehabilitation Engineering, Peking University, Beijing</wicri:regionArea>
<placeName><settlement type="city">Pékin</settlement>
</placeName>
<orgName type="university">Université de Pékin</orgName>
</affiliation>
</author>
</analytic>
<series><title level="j">PLoS ONE</title>
<idno type="eISSN">1932-6203</idno>
<imprint><date when="2013">2013</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en"><p>We often need to learn how to move based on a single performance measure that reflects the overall success of our movements. However, movements have many properties, such as their trajectories, speeds and timing of end-points, thus the brain needs to decide which properties of movements should be improved; it needs to solve the credit assignment problem. Currently, little is known about how humans solve credit assignment problems in the context of reinforcement learning. Here we tested how human participants solve such problems during a trajectory-learning task. Without an explicitly-defined target movement, participants made hand reaches and received monetary rewards as feedback on a trial-by-trial basis. The curvature and direction of the attempted reach trajectories determined the monetary rewards received in a manner that can be manipulated experimentally. Based on the history of action-reward pairs, participants quickly solved the credit assignment problem and learned the implicit payoff function. A Bayesian credit-assignment model with built-in forgetting accurately predicts their trial-by-trial learning.</p>
</div>
</front>
<back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Kaelbling, Lp" uniqKey="Kaelbling L">LP Kaelbling</name>
</author>
<author><name sortKey="Littman, Ml" uniqKey="Littman M">ML Littman</name>
</author>
<author><name sortKey="Moore, Aw" uniqKey="Moore A">AW Moore</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Houk, Jc" uniqKey="Houk J">JC Houk</name>
</author>
<author><name sortKey="Buckingham, J" uniqKey="Buckingham J">J Buckingham</name>
</author>
<author><name sortKey="Barto, Ag" uniqKey="Barto A">AG Barto</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Berniker, M" uniqKey="Berniker M">M Berniker</name>
</author>
<author><name sortKey="Kording, K" uniqKey="Kording K">K Körding</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Schultz, W" uniqKey="Schultz W">W Schultz</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Todorov, E" uniqKey="Todorov E">E Todorov</name>
</author>
<author><name sortKey="Jordan, Mi" uniqKey="Jordan M">MI Jordan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wolpert, Dm" uniqKey="Wolpert D">DM Wolpert</name>
</author>
<author><name sortKey="Ghahramani, Z" uniqKey="Ghahramani Z">Z Ghahramani</name>
</author>
<author><name sortKey="Flanagan, Jr" uniqKey="Flanagan J">JR Flanagan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Scholz, Jp" uniqKey="Scholz J">JP Scholz</name>
</author>
<author><name sortKey="Schoner, G" uniqKey="Schoner G">G Schoner</name>
</author>
<author><name sortKey="Latash, Ml" uniqKey="Latash M">ML Latash</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Muller, H" uniqKey="Muller H">H Müller</name>
</author>
<author><name sortKey="Sternad, D" uniqKey="Sternad D">D Sternad</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Pennington, N" uniqKey="Pennington N">N Pennington</name>
</author>
<author><name sortKey="Hastie, R" uniqKey="Hastie R">R Hastie</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Latash, Ml" uniqKey="Latash M">ML Latash</name>
</author>
<author><name sortKey="Scholz, Jp" uniqKey="Scholz J">JP Scholz</name>
</author>
<author><name sortKey="Sch Ner, G" uniqKey="Sch Ner G">G Sch ner</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Scholz, Jp" uniqKey="Scholz J">JP Scholz</name>
</author>
<author><name sortKey="Schoner, G" uniqKey="Schoner G">G Schoner</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Muller, H" uniqKey="Muller H">H Müller</name>
</author>
<author><name sortKey="Sternad, D" uniqKey="Sternad D">D Sternad</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Sternad, D" uniqKey="Sternad D">D Sternad</name>
</author>
<author><name sortKey="Abe, Mo" uniqKey="Abe M">MO Abe</name>
</author>
<author><name sortKey="Hu, X" uniqKey="Hu X">X Hu</name>
</author>
<author><name sortKey="Muller, H" uniqKey="Muller H">H Müller</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Martin, Ta" uniqKey="Martin T">TA Martin</name>
</author>
<author><name sortKey="Norris, Sa" uniqKey="Norris S">SA Norris</name>
</author>
<author><name sortKey="Greger, Be" uniqKey="Greger B">BE Greger</name>
</author>
<author><name sortKey="Thach, Wt" uniqKey="Thach W">WT Thach</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Kudo, K" uniqKey="Kudo K">K Kudo</name>
</author>
<author><name sortKey="Tsutsui, S" uniqKey="Tsutsui S">S Tsutsui</name>
</author>
<author><name sortKey="Ishikura, T" uniqKey="Ishikura T">T Ishikura</name>
</author>
<author><name sortKey="Ito, T" uniqKey="Ito T">T Ito</name>
</author>
<author><name sortKey="Yamamoto, Y" uniqKey="Yamamoto Y">Y Yamamoto</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Latash, Ml" uniqKey="Latash M">ML Latash</name>
</author>
<author><name sortKey="Scholz, Jp" uniqKey="Scholz J">JP Scholz</name>
</author>
<author><name sortKey="Sch Ner, G" uniqKey="Sch Ner G">G Sch ner</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Fu, Wt" uniqKey="Fu W">WT Fu</name>
</author>
<author><name sortKey="Anderson, Jr" uniqKey="Anderson J">JR Anderson</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Fu, Wt" uniqKey="Fu W">WT Fu</name>
</author>
<author><name sortKey="Anderson, Jr" uniqKey="Anderson J">JR Anderson</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Houk, Jc" uniqKey="Houk J">JC Houk</name>
</author>
<author><name sortKey="Buckingham, Jt" uniqKey="Buckingham J">JT Buckingham</name>
</author>
<author><name sortKey="Barto, Ag" uniqKey="Barto A">AG Barto</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wei, K" uniqKey="Wei K">K Wei</name>
</author>
<author><name sortKey="Kording, K" uniqKey="Kording K">K Kording</name>
</author>
</analytic>
</biblStruct>
</listBibl>
</div1>
</back>
</TEI>
<pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">PLoS One</journal-id>
<journal-id journal-id-type="iso-abbrev">PLoS ONE</journal-id>
<journal-id journal-id-type="publisher-id">plos</journal-id>
<journal-id journal-id-type="pmc">plosone</journal-id>
<journal-title-group><journal-title>PLoS ONE</journal-title>
</journal-title-group>
<issn pub-type="epub">1932-6203</issn>
<publisher><publisher-name>Public Library of Science</publisher-name>
<publisher-loc>San Francisco, USA</publisher-loc>
</publisher>
</journal-meta>
<article-meta><article-id pub-id-type="pmid">23408972</article-id>
<article-id pub-id-type="pmc">3568147</article-id>
<article-id pub-id-type="publisher-id">PONE-D-12-24681</article-id>
<article-id pub-id-type="doi">10.1371/journal.pone.0055352</article-id>
<article-categories><subj-group subj-group-type="heading"><subject>Research Article</subject>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Biology</subject>
<subj-group><subject>Anatomy and Physiology</subject>
<subj-group><subject>Neurological System</subject>
<subj-group><subject>Motor Systems</subject>
</subj-group>
</subj-group>
</subj-group>
<subj-group><subject>Neuroscience</subject>
<subj-group><subject>Learning and Memory</subject>
<subject>Motor Systems</subject>
</subj-group>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Social and Behavioral Sciences</subject>
<subj-group><subject>Psychology</subject>
<subj-group><subject>Cognitive Psychology</subject>
<subj-group><subject>Learning</subject>
<subject>Memory</subject>
</subj-group>
</subj-group>
<subj-group><subject>Psychophysics</subject>
</subj-group>
</subj-group>
</subj-group>
</article-categories>
<title-group><article-title>Credit Assignment during Movement Reinforcement Learning</article-title>
<alt-title alt-title-type="running-head">Movement Reinforcement Learning</alt-title>
</title-group>
<contrib-group><contrib contrib-type="author"><name><surname>Dam</surname>
<given-names>Gregory</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Kording</surname>
<given-names>Konrad</given-names>
</name>
<xref ref-type="aff" rid="aff2"><sup>2</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Wei</surname>
<given-names>Kunlin</given-names>
</name>
<xref ref-type="aff" rid="aff3"><sup>3</sup>
</xref>
<xref ref-type="corresp" rid="cor1"><sup>*</sup>
</xref>
</contrib>
</contrib-group>
<aff id="aff1"><label>1</label>
<addr-line>Department of Behavioral Sciences, University of Rio Grande, Rio Grande, Ohio, United States of America</addr-line>
</aff>
<aff id="aff2"><label>2</label>
<addr-line>Department of Physical Medicine and Rehabilitation, Rehabilitation Institute of Chicago, Northwestern University, Chicago, Illinois, United States of America</addr-line>
</aff>
<aff id="aff3"><label>3</label>
<addr-line>Department of Psychology, Key Laboratory of Machine Perception (Ministry of Education), Beijing Engineering Research Center of Intelligent Rehabilitation Engineering, Peking University, Beijing, China</addr-line>
</aff>
<contrib-group><contrib contrib-type="editor"><name><surname>Gribble</surname>
<given-names>Paul L.</given-names>
</name>
<role>Editor</role>
<xref ref-type="aff" rid="edit1"></xref>
</contrib>
</contrib-group>
<aff id="edit1"><addr-line>The University of Western Ontario, Canada</addr-line>
</aff>
<author-notes><corresp id="cor1">* E-mail: <email>wei.kunlin@pku.edu.cn</email>
</corresp>
<fn fn-type="conflict"><p><bold>Competing Interests: </bold>
The authors have declared that no competing interests exist.</p>
</fn>
<fn fn-type="con"><p>Conceived and designed the experiments: GD KK. Performed the experiments: GD. Analyzed the data: KW GD. Contributed reagents/materials/analysis tools: KK. Wrote the paper: KW GD KK.</p>
</fn>
</author-notes>
<pub-date pub-type="collection"><year>2013</year>
</pub-date>
<pub-date pub-type="epub"><day>8</day>
<month>2</month>
<year>2013</year>
</pub-date>
<volume>8</volume>
<issue>2</issue>
<elocation-id>e55352</elocation-id>
<history><date date-type="received"><day>15</day>
<month>8</month>
<year>2012</year>
</date>
<date date-type="accepted"><day>30</day>
<month>12</month>
<year>2012</year>
</date>
</history>
<permissions><copyright-year>2013</copyright-year>
<copyright-holder>Dam et al</copyright-holder>
<license><license-p>This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.</license-p>
</license>
</permissions>
<abstract><p>We often need to learn how to move based on a single performance measure that reflects the overall success of our movements. However, movements have many properties, such as their trajectories, speeds and timing of end-points, thus the brain needs to decide which properties of movements should be improved; it needs to solve the credit assignment problem. Currently, little is known about how humans solve credit assignment problems in the context of reinforcement learning. Here we tested how human participants solve such problems during a trajectory-learning task. Without an explicitly-defined target movement, participants made hand reaches and received monetary rewards as feedback on a trial-by-trial basis. The curvature and direction of the attempted reach trajectories determined the monetary rewards received in a manner that can be manipulated experimentally. Based on the history of action-reward pairs, participants quickly solved the credit assignment problem and learned the implicit payoff function. A Bayesian credit-assignment model with built-in forgetting accurately predicts their trial-by-trial learning.</p>
</abstract>
<funding-group><funding-statement>This work has been funded by NIH Grants R01NS-063399 and R01NS-057814, the National Natural Science Foundation of China (No. 31000456, and 61020106005), the Fundamental Research Funds for the Central Universities, and the Collaboration Funds from Beijing Municipal Education Commission. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</funding-statement>
</funding-group>
<counts><page-count count="8"></page-count>
</counts>
</article-meta>
</front>
<body><sec id="s1"><title>Introduction</title>
<p>When a golfer decides to swing, the movement decision involves judging, among other things, the target distance, wind strength and direction, and how to handle the chosen club. The error in the resulting trajectory and the end position of the ball has many fewer dimensions than the golfer’s swing decision. In fact, most movement decisions we face have many choice dimensions that determine the final outcome. Often the feedback we obtain regarding the success of our decisions is low-dimension and may not tell how well we did with regard to every choice dimension. In such many-to-one mapping situations efficient learning should involve discovering what properties of the movement contributed to error and correcting them accordingly in order to achieve the desired outcomes.</p>
<p>Learning what property of a decision has resulted in an error is referred to as the <italic>credit assignment problem</italic>
in the domains of machine learning and neurobiology <xref ref-type="bibr" rid="pone.0055352-Kaelbling1">[1]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Houk1">[2]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Berniker1">[3]</xref>
. However, little is understood about the behavioral strategies people use to solve the credit assignment problem during movement decision-making <xref ref-type="bibr" rid="pone.0055352-Schultz1">[4]</xref>
. For efficient motor learning, it is necessary to discover the important movement properties and optimizing them to maximize rewards. This can be difficult due to the aforementioned many-to-one mapping between movement properties and movement outcome. More often than not, the results of movement decisions may not speak directly to each individual movement property. When a basketball player makes a jump shot, a short shot to the basket might be caused by an early release of the ball, insufficient acceleration of the wrist rotation, or lower-than-expected jump height, or a combination of all these movement properties. The challenge left to the player is to determine what properties of the movement decision should be blamed for a failure.</p>
<p>The many-to-one mapping between movement properties and movement outcome, traditionally referred to as the degrees of freedom problem, has been extensively investigated in motor control <xref ref-type="bibr" rid="pone.0055352-Bernstein1">[5]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Todorov1">[6]</xref>
. Previous studies put emphasis on how different movement properties are coordinated and controlled to produce a desired movement goal. However, few studies have been conducted, from the perspective of reinforcement learning, on the reverse yet mathematically equivalent problem <xref ref-type="bibr" rid="pone.0055352-Wolpert1">[7]</xref>
of how the nervous system learns to differentially adjust different properties of a movement based on limited reward information. This type of credit assignment problem can be conveniently investigated in goal-directed movement tasks such as throwing or shooting <xref ref-type="bibr" rid="pone.0055352-Scholz1">[8]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Mller1">[9]</xref>
. In these tasks a combination of movement properties can uniquely determine a single performance score. This opens the window for us to quantitatively vary reward functions and to examine whether and how people assign credits to different movement properties during movement reinforcement learning.</p>
<p>Here we investigate how people solve the credit assignment problem while learning a motor skill with reinforcement. Participants were asked to match their reaching movements to invisible target trajectories that varied in both direction and curvature. These two movement properties were differently weighted to determine the monetary reward that was presented at the end of each trial. Across two different groups of participants we varied the weighting of movement direction and curvature to the rewards. By examining the differences in learning strategies between groups we were able to investigate how participants learn to assign credit to different movement properties. We find that participants’ exploratory strategies are sensitive to the weighing of reward functions, suggesting that they are able to solve the credit assignment problem quickly with limited feedback. Furthermore participants use reward information efficiently in trial-by-trial exploration behaviors, in a manner consistent with a Bayes-like strategy.</p>
</sec>
<sec sec-type="methods" id="s2"><title>Methods</title>
<sec id="s2a"><title>The Experiment</title>
<p>Experimental techniques were approved in accordance with Northwestern University’s Policy statement on the use of humans in experiments and with federal guidelines. Informed written consent was obtained from sixteen right-handed, healthy participants (five males; mean age of 28.14 years (SD = 5.07)). Participants were seated in a chair facing a computer screen (<xref ref-type="fig" rid="pone-0055352-g001">Fig. 1A</xref>
). They were asked to draw trajectories with their dominant hand using the stylus of a PHANToM Premium 1.0 haptic robot (SensAble Technologies, Inc., Woburn, MA). Trajectories were made by sliding the tip of the robot’s stylus along the surface of a desk. With the stylus, participants controlled the position of a cursor on the computer screen. Each participant attempted to match 50 different, invisible target trajectories that varied randomly in both direction and curvature. For each target trajectory participants made 25 successive attempts to approximate as closely as possible each desired target trajectory, resulting in a total of 1250 movements for the experiment. All movements started from the same position at the center of workspace, 10 cm from the table edge in front of the subject. The reaches ended when the cursor reached a displayed line 7 cm in front of (Y-direction) from the starting position. A beep was played by computer speakers to indicate the end of the trial. The shape (X and Y coordinates) of the target trajectory was defined by<disp-formula id="pone.0055352.e001"><graphic xlink:href="pone.0055352.e001"></graphic>
<label>(1)</label>
</disp-formula>
where the x-coordinates (<inline-formula><inline-graphic xlink:href="pone.0055352.e002.jpg"></inline-graphic>
</inline-formula>
) of the target trajectory are determined by two parameters, direction (α) and curvature (β). For each target trajectory, these two parameters were chosen randomly within the interval [−1, 1] with a fixed incremental size of 0.1. As a result, all target trajectories were confined in a quadrangular table-top space of 21.7 cm in width and 7 cm in depth. Note the Phantom robot’s measuring range is 25.4 cm wide and 12.7 cm deep. The stylus is also light-weighted with small inertial. Thus, neither the participants’ biomechanics, nor the mechanical constraints of the robot are likely to limit the ability to accurately match the target trajectories.</p>
<fig id="pone-0055352-g001" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0055352.g001</object-id>
<label>Figure 1</label>
<caption><title>The experimental setup.</title>
<p>A): The cartoon illustration of the setup. Participants sat before a desk and made movement trajectories on the horizontal desktop with a hand-held stylus. The hand displacement was registered by a robot. The feedback was provided via a computer monitor placed on the desk. B): A graphical representation of how trajectories were varied in both direction (α) and curvature (β). C): The learning progress of matching trajectories to a hidden target trajectory within a session of 25 trials.</p>
</caption>
<graphic xlink:href="pone.0055352.g001"></graphic>
</fig>
<p>Each attempted trajectory resulted in a monetary score that provided participants with information on how closely the hidden target trajectory was matched. The score was shown on the top-left corner of a computer screen after each trial. The score had a maximum achievable amount of 50 cents. Participants were informed that they would receive the highest monetary reward they achieved from the 25 trajectory attempts for each target trajectory. Received monetary amounts were summed for all 50 target trajectories making 25$ the maximum achievable participation stipend. To determine the score, the position of the cursor was recorded at an approximate sampling frequency of 250 Hz. The participants’ attempted trajectory was then fitted to eq.1 to obtain the estimates of α and β. The magnitude of the monetary reward was determined using the difference between the estimated and the target α and β. These two errors were weighted differently for different experimental conditions.</p>
<p>Participants were split into two groups with eight participants in each group. One group learned target trajectories where the rewards were computed using a reward function with a large weight on the α parameter:<disp-formula id="pone.0055352.e003"><graphic xlink:href="pone.0055352.e003"></graphic>
<label>(2)</label>
</disp-formula>
where Δα and Δβ are the normalized, absolute error for direction and curvature, respectively. They will usually take the values between 0 and 1 if the actual direction and curvature are within the range of target trajectories. <italic>W</italic>
and <italic>w</italic>
stand for large and small weights for penalizing the direction and curvature error, respectively. They were set at 0.8 and 0.2 in this condition. The resulting <inline-formula><inline-graphic xlink:href="pone.0055352.e004.jpg"></inline-graphic>
</inline-formula>
will then be multiplied by 50 cents as the actual monetary reward shown at the end of each trial. The effect of this differential weight is that that errors in the direction of attempted trajectories lead to larger reductions in obtained monetary rewards than errors in the curvature of trajectories. In rare cases the curvature and direction that participants performed with can exceed the range of target trajectories, resulting in a large error and a negative reward value. However, we only displayed a zero-reward for these trials.</p>
<p>The other eight participants learned target trajectories while provided with feedback using a heavily weighted β parameter of the reward function:<disp-formula id="pone.0055352.e005"><graphic xlink:href="pone.0055352.e005"></graphic>
<label>(3)</label>
</disp-formula>
Where the curvature of the attempted trajectory has a larger weight <italic>W</italic>
and the direction has a smaller weight <italic>w</italic>
. The two conditions associated with these two reward functions were termed α-reward and β-reward conditions, respectively.</p>
<p>One of the unique affordances of this experimental design is that the composite reward function above can be weighted differently for each dimension of movement (direction or curvature). By suppressing or augmenting the consequences of errors made in each movement dimension individually, we can study people’s behavioral structural credit during movement reward learning.</p>
</sec>
<sec id="s2b"><title>The Bayesian Model</title>
<p>A good learning strategy would efficiently explore different movement properties while maintaining a memory of past movements and their associated rewards. Within a decision theoretic framework such a strategy can be formalized using Bayesian decision theory. The optimal decision to approximate the target trajectory combines the best <italic>present</italic>
estimate (likelihood) with information about <italic>past</italic>
(prior) trajectories as shown in the following equation:<disp-formula id="pone.0055352.e006"><graphic xlink:href="pone.0055352.e006"></graphic>
<label>(4)</label>
</disp-formula>
</p>
<p>At any trial <italic>t</italic>
, the prior is defined as the probability that the past rewards (<italic>r<sub>1:</sub>
</italic>
<sub>t−1</sub>
) and attempted trajectory (<italic>s<sub>1:</sub>
</italic>
<sub>t−1</sub>
) would have been observed under the assumption of a given target (<italic>target</italic>
<sub>α,β</sub>
= <italic>pos</italic>
<sub>α,β</sub>
). The likelihood probability is defined as the probability of the current trajectory and reward (<italic>s<sub>t</sub>
,r<sub>t</sub>
</italic>
) given the target shape (<italic>target</italic>
<sub>α,β</sub>
= <italic>pos</italic>
<sub>α,β</sub>
). Furthermore, it is known that peoples’ memory of past decisions is less than perfect on such tasks <xref ref-type="bibr" rid="pone.0055352-Pennington1">[10]</xref>
. To incorporate this a memory decay term (bounded in [0,1]) was added. Finally, the probability that the target trajectory (<italic>target</italic>
<sub>α,β</sub>
) has a given shape (<italic>pos</italic>
<sub>α,β</sub>
) at trial <italic>t</italic>
is equal to probability of combining the prior, the likelihood and the memory decay.</p>
<p>For implementing Bayesian updating, we assume participants maintain a probability map of the to-be-learned curvature and direction, and iteratively update that map after receiving the reward upon each trial (<xref ref-type="fig" rid="pone-0055352-g002">Fig. 2</xref>
). The map codes the probability of each curvature-direction combination and decays linearly in between trials as captured by the memory decay term λ:<disp-formula id="pone.0055352.e007"><graphic xlink:href="pone.0055352.e007"></graphic>
<label>(5)</label>
</disp-formula>
</p>
<fig id="pone-0055352-g002" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0055352.g002</object-id>
<label>Figure 2</label>
<caption><title>The implementation of the Bayesian model.</title>
<p>The implementation assumes a two-dimension probability map that is updated iteratively trial by trial. It is a 200×200 matrix to code the probability of each α-β combination. Each value in the matrix is normalized such that the sum of all possibilities on the map equals 1. The pink cross denotes the current target direction and curvature. The gray dot denotes the best solution before the current trial <italic>t</italic>
and the black dot denotes the best solution after finishing the current trial. The map from a previous trial is degraded by memory decay and it then serves as the prior before the current trial. The prediction error, the difference between the predicted reward based on the direction and curvature used in the current trial and the actual received reward, serves as likelihood distribution to update the probability map. By combining the prior and the likelihood, the probability map is updated to form the posterior distribution. The learning is demonstrated in that the best solution of the posterior, compared to that of the prior, becomes closer to the target solution. The data are from a typical trial (the 4<sup>th</sup>
trial in a 25 learning sequence) from a single participant.</p>
</caption>
<graphic xlink:href="pone.0055352.g002"></graphic>
</fig>
<p>Where <inline-formula><inline-graphic xlink:href="pone.0055352.e008.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0055352.e009.jpg"></inline-graphic>
</inline-formula>
are the probability maps updated at the previous trial and before the current trial, respectively. We further assume that participants implicitly approximate the reward function by giving weights to errors in curvature and in direction. Hence, the predicted reward, <inline-formula><inline-graphic xlink:href="pone.0055352.e010.jpg"></inline-graphic>
</inline-formula>
, is computed as</p>
<p><disp-formula id="pone.0055352.e011"><graphic xlink:href="pone.0055352.e011"></graphic>
</disp-formula>
Where <inline-formula><inline-graphic xlink:href="pone.0055352.e012.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0055352.e013.jpg"></inline-graphic>
</inline-formula>
are the squared errors in both dimensions on a trial. This predicted reward is compared with the actual reward achieved on that trial (<inline-formula><inline-graphic xlink:href="pone.0055352.e014.jpg"></inline-graphic>
</inline-formula>
) and the discrepancy (prediction error) is then used to update the probability map with a constant learning rate <inline-formula><inline-graphic xlink:href="pone.0055352.e015.jpg"></inline-graphic>
</inline-formula>
:</p>
<p><disp-formula id="pone.0055352.e016"><graphic xlink:href="pone.0055352.e016"></graphic>
</disp-formula>
Where <inline-formula><inline-graphic xlink:href="pone.0055352.e017.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0055352.e018.jpg"></inline-graphic>
</inline-formula>
are the probability map before and after the current trial, respectively. Thus the model has 4 free parameters: the memory decay term <inline-formula><inline-graphic xlink:href="pone.0055352.e019.jpg"></inline-graphic>
</inline-formula>
, weights for errors <inline-formula><inline-graphic xlink:href="pone.0055352.e020.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0055352.e021.jpg"></inline-graphic>
</inline-formula>
, and the learning constant <inline-formula><inline-graphic xlink:href="pone.0055352.e022.jpg"></inline-graphic>
</inline-formula>
. They were estimated by fitting the model to all 50 learning sequences for each participant.</p>
</sec>
</sec>
<sec id="s3"><title>Results</title>
<p>Here we designed a reinforcement learning task requiring the actor to assign varying credits to different movement properties based on single monetary rewards. By assessing trial-by-trial exploration behavior, we can examine whether participants can solve this credit assignment problem efficiently and whether reward-movement pairings are sufficient to support this learning.</p>
<p>We first examined whether participants could learn the hidden target trajectory with monetary rewards as feedback (<xref ref-type="fig" rid="pone-0055352-g003">Fig. 3</xref>
). Data from a typical participant in the α-reward condition indicates that the movement trajectories incrementally match the hidden target trajectory in terms of the direction and the curvature (<xref ref-type="fig" rid="pone-0055352-g003">Fig. 3A</xref>
). For this particular participant, the learning of two trajectory properties reaches a plateau within 25 trials. However, not all the target trajectories are learned equally well. A typical participant from the β-reward condition exhibits improvement in trajectory curvature early on but fails to minimize direction error within 25 trials (<xref ref-type="fig" rid="pone-0055352-g003">Fig. 3B</xref>
).</p>
<fig id="pone-0055352-g003" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0055352.g003</object-id>
<label>Figure 3</label>
<caption><title>Learning data from typical participants from the α-reward condition (upper panels, A) and the β-reward condition (lower panels, B).</title>
<p>A): Five individual trial trajectories (blue) along with its corresponding hidden target trajectory (red) are shown in separate panels from left to right. The rightmost panel displays the absolute error in trajectory direction (α) and curvature (β). The direction of targets is learned earlier than curvature. B): Panels follow the same format as the upper panels. Curvature learning occurs early but the errors in the direction of targets remain high throughout the session.</p>
</caption>
<graphic xlink:href="pone.0055352.g003"></graphic>
</fig>
<p>A group analysis reveals a similar picture. For both reward conditions, participants exhibit exponential learning and their performance (in terms of rewards received) asymptotes around the 20<sup>th</sup>
trial (<xref ref-type="fig" rid="pone-0055352-g004">Fig. 4A</xref>
). The average highest monetary reward achieved across participants and across all 50 target trajectories is 45.6 cents (SD = 3.8), which is very close to the maximum possible reward of 50 cents. Participants demonstrate better initial performance in the α-reward condition, perhaps suggesting that the reach direction is inherently easier to learn than its curvature. The learning curves of the reward from the two conditions are similar but those of two movement properties are different (see below). Overall, these results suggest that participants are very adept at learning new target trajectories in 25 attempts.</p>
<fig id="pone-0055352-g004" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0055352.g004</object-id>
<label>Figure 4</label>
<caption><title>Learning-related changes.</title>
<p>A): The monetary rewards, averaged over all target trajectories, are plotted as a function of trials. The two shaded curves stand for mean±SEM over participants for two reward conditions separately. B): The learning curves of two trajectory properties, direction (α) and curvature (β), are plotted in black and red, respectively. The green and the gray lines denote their corresponding exponential fits. The results from the two reward conditions are presented in two separate subplots. C): The same learning curves from B) are re-plotted in the α-β space. The arrows indicate directions of changes and their sizes are proportional to magnitude of changes.</p>
</caption>
<graphic xlink:href="pone.0055352.g004"></graphic>
</fig>
<p>If one of the movement properties was weighted more in the rewards, we would expect that participants should preferentially explore that property. This is indeed the case (<xref ref-type="fig" rid="pone-0055352-g004">Fig. 4B</xref>
). The error in the weighted property is corrected more rapidly. By fitting an exponential function to the average learning curve of each property, we find that the decay constant in the α-reward condition is 3.3 and 8.1 for α and β, respectively. In contrast, in the β-reward condition it is 11.8 and 3.7 for α and β, respectively. These results indicate that the weighted movement property is learned faster than the un-weighted one. Participants also settle at different asymptotes after 25-trial learning in two conditions: for the α-reward condition both α and β errors settles at 0.23, while for the β-reward condition α error settles at 0.48 and β error at 0.15. In the β-reward condition, α error remains high late into the learning sequence even though the reward reaches a plateau. In sum, the learning rate of the weighted property is substantially faster than that of the less-weighted despite that learning of the two properties is completed to different extents.</p>
<p>The preferential improvement in the weighted property can be further demonstrated by casting the two learning curves into a two-error-dimension space (<xref ref-type="fig" rid="pone-0055352-g004">Fig. 4C</xref>
). Learning happens predominantly in direction of the trajectory when the reward function is α (direction) weighted and predominantly in curvature when the reward function is β (curvature) weighted. These results suggest that participants are able to infer the most important dimension in the movement learning task based on a feedback history of uni-dimensional monetary rewards.</p>
<p>In order to understand how participants used information about trajectory and rewards, a Bayesian model was developed and fitted to the participants’ movement data. The average changes in trajectory properties between subsequent attempts are shown in a quiver plot (<xref ref-type="fig" rid="pone-0055352-g005">Fig. 5</xref>
). Vectors are sparse as only a limited set of possible transitions are observed among our participants. Those long vectors, indicating large corrections over successive trials, mostly appear when large curvature and/or direction errors occur. Some trials resulted in small or even negative rewards but they constitute only a small fraction of total trials (1.19% and 0.09% for the α- and β-reward conditions, respectively). Note that zero is the lowest reward feedback given on any trials. Overall, the Bayesian model captures the trial-by-trial changes in trajectory properties, shown by the close match between data and model vectors in the terms of direction and magnitude. The Bayesian model explains 42.9% and 49.6% of total variance in participants’ trajectories decisions for the α- and β-reward conditions, respectively. This suggests that participants might employ a near optimal strategy for updating information about trajectory shape and rewards, and use this knowledge to inform subsequent trajectory decisions. Interestingly, the memory decay term is 0.32±0.1 and 0.54±0.1 for α- and β- reward conditions, respectively. This suggests that participants relied on only a fraction of past information to guide the current search.</p>
<fig id="pone-0055352-g005" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0055352.g005</object-id>
<label>Figure 5</label>
<caption><title>Quiver plot of average changes in movement properties with a α- (A) and a β- (B) reward function.</title>
<p>The x- and y-axis denote the absolute errors in two movement properties, respectively. The background temperature plots display the reward functions with the maximum achievable reward centered at zero alpha-beta error. The higher the temperature the higher the reward. White vectors are average parameter changes from one trial to the next across all target trajectories and all participants. Black vectors represent the Bayesian model’s predictions.</p>
</caption>
<graphic xlink:href="pone.0055352.g005"></graphic>
</fig>
<p>Is it possible that the ability of matching invisible targets was improved gradually over the course of experiment? We test this possible meta-learning effect by evaluating learning curves for different parts of the experiments. The total 50 target trajectories are grouped into 5 sessions, 10 targets each. The learning curves, based on α error, β error and rewards are shown separately (<xref ref-type="fig" rid="pone-0055352-g006">Fig. 6</xref>
). We quantified the amount of learning by taking the average of the last 5 trials and compared them across sessions. For the α-reward condition, there is no significantly difference in time for all three variables despite an overall improving trend during the course of experiment. For the β-reward condition, the amount of learning is significantly larger in terms of β error and rewards between the first session and the last sessions (post-hoc Tukey tests, <italic>p</italic>
<0.05). Taken together, a meta-learning effect is present but it is confined to the initial learning stages during the β-reward condition only.</p>
<fig id="pone-0055352-g006" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0055352.g006</object-id>
<label>Figure 6</label>
<caption><title>Average learning curves over each of 10 target trajectories are shown for the α-reward condition (A) and the β-reward condition (B) separately.</title>
<p>Displayed in different panels, learning curves are based on α error, β error and actual monetary rewards.</p>
</caption>
<graphic xlink:href="pone.0055352.g006"></graphic>
</fig>
</sec>
<sec id="s4"><title>Discussion</title>
<p>We have examined how people learn to move to match invisible target trajectories with only single monetary rewards as feedback after each movement. We show that participants solve this credit assignment problem quickly and converge to the correct solution despite the many-to-one mapping of the learning task. Their search for the target is systematically affected by the properties of the reward function used as demonstrated by faster learning of the more rewarded movement property. Participants learn quickly what movement properties matter most. Their trial-by-trial learning is consistent with the predictions of a Bayesian model, suggesting that they efficiently use reinforcement information during learning.</p>
<p>Even though participants quickly and successfully discovered the heavily-weighted error dimension, the two movement properties were not learned equally. Specifically, in the β-reward condition participants failed to optimize the direction error to the same degree as that of the curvature error, even though they were apparently capable of doing so as shown in the α-reward condition (<xref ref-type="fig" rid="pone-0055352-g004">Fig. 4</xref>
). At the same time, despite of this relatively large direction error asymptote, similarly good rewards were achieved by optimizing the curvature. Thus, this large asymptote might reflect that importance of reward feedback <xref ref-type="bibr" rid="pone.0055352-Kaelbling2">[11]</xref>
: participants are slow in optimizing less-penalized dimension when reward feedback is already improved to a plateau. It is interesting that in the α-reward condition the curvature error, the less-penalized dimension, did not exhibit similarly slack learning. This difference thus highlights that less-penalized movement properties are not optimized equally fast when only overall reward feedback is provided. This is interesting since when the same two movement properties are heavily-weighted their learning was equally fast. The system is thus efficient in terms of preferentially dealing with more important control variables as the minimum-intervention principle suggests <xref ref-type="bibr" rid="pone.0055352-Todorov1">[6]</xref>
.</p>
<p>Our findings show that search behavior in reinforcement motor learning is shaped by the properties of the reward function as the more rewarding dimension is preferentially learned. Analogous findings has been demonstrated when analyzing the control of movements where a reduction in variability of the more performance-affected dimension is observed <xref ref-type="bibr" rid="pone.0055352-Mller1">[9]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Latash1">[12]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Scholz2">[13]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Mller2">[14]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Sternad1">[15]</xref>
. In these studies, the relationship between control variables (e.g., joint angles) and a performance measure (e.g., the endpoint precision of a reach movement) have been characterized by certain analytic functions termed solution manifolds. Often times, the interactions between various control variables are of particular importance for good performance, as prescribed by the structure of a solution manifold. A universal finding in these studies is that people control their movements so as to minimize the variability of the critical dimensions as such inherent motor variability will minimally impact performance {Martin, 2002 #1303;Kudo, 2000 #1304}. These observed covariations between control variables reflect the established synergy <xref ref-type="bibr" rid="pone.0055352-Latash2">[18]</xref>
or strategies based on optimal feedback control <xref ref-type="bibr" rid="pone.0055352-Todorov1">[6]</xref>
. Conceptually, the solution manifold is equivalent to the reward function in the present study and both of them map movement properties to a performance measure. Our findings thus suggest that <italic>before</italic>
a stable performance strategy is formed the nervous system <italic>learns</italic>
to first modify the movement properties that matter the most for movement performance. This preferential error reduction is flexible and dynamically adjusted according to task demands that are experimentally varied with the use of different reward functions.</p>
<p>Some effort has been devoted towards studying the <italic>temporal</italic>
credit assignment problem in reinforcement learning of motor skill. In these tasks, an agent needs to judge <italic>when</italic>
was an action that resulted in error committed during the execution of a sequence of actions <xref ref-type="bibr" rid="pone.0055352-Sutton1">[19]</xref>
. Accordingly, these studies use action sequence learning tasks such as maze-searching {Tolman, 1930 #1403;Fu, 2006 #1404}, sequential choice task {Fu, 2008 #1402} and mine-field navigation {Gordon, 1994 #1401}. The tasks are designed to test how credits are assigned to actions in multistep choice situations. Our study instead addresses the <italic>structural</italic>
credit assignment problem <xref ref-type="bibr" rid="pone.0055352-Gullapalli1">[24]</xref>
, <xref ref-type="bibr" rid="pone.0055352-Houk2">[25]</xref>
, focusing on how the agent assigns credits to different properties of a single action. The agent needs to learn <italic>what</italic>
property of a decision lead to error in order to correct it, an important problem that has received little attention. This work represents an initial endeavor to study the structural credit assignment problem within the domain of motor learning.</p>
<p>The close match between participants’ movement data and the predictions of a Bayesian model suggests that people efficiently integrate past and present information to inform subsequent movement decisions. Our Bayesian model is conceptually similar to other reinforcement learning models where the decision for the next attempt is made on the basis of the history of reward-movement pairs. An interesting finding is that the memory decay between successive attempts was fairly high (0.32 and 0.54, where 1 represents perfect recall). Trial-by-trial search behavior has a heavy temporal discounting component and past action-reward pairs have limited influence on the current decision. This suggest that either people have poor recall of past action-reward parings, or that they are over-confident of their current exploratory strategies, neglecting information gained from previous exploration.</p>
<p>We acknowledge that our model is simplified by assuming constant memory decay and learning rate over the whole learning process. The assumption made is that learning is essentially the same over the course of the experiment. Our results indicate that there is some evidence of meta-learning in the β-reward condition where the learning is improved after exposure to 10 target trajectories. Future modeling efforts could incorporate small meta-learning effects to capture the data better. On the other hand, to our best knowledge there is no good alternative model for capturing the trial-to-trial learning for this type of structural credit assignment task even though similar tasks have been investigated in motor learning {Müller, 2004 #892;Scholz, 1999 #1300}.</p>
<p>Credit assignment problems also arise in motor adaptation where the nervous system adapts movement to control perturbations from external environment or from noise in the movement system itself. It has been proposed that adaptation depends on probabilistic inference of causality of these changes: movement perturbations that are less likely to be caused by changes in motor apparatus are less likely to be learned and generalized <xref ref-type="bibr" rid="pone.0055352-Berniker1">[3]</xref>
. The movement errors that are less likely caused by the actor and therefore more likely caused by external perturbations are also less likely to be corrected <xref ref-type="bibr" rid="pone.0055352-Wei1">[26]</xref>
. These studies suggest that the nervous system assigns different weights to movement error or perturbation according to the inferred causal structure of error source during motor adaptation. In the present study, all errors are attributed to the actor who is able to efficiently learn what properties matter most with regards to the desired goal. In the reinforcement task used in our experiment, no single action-reward pair could reveal the structure of the payoff function. Remarkably, participants are able to quickly glean the pay-off function by exploring different movement properties and noticing how they affect the magnitude of rewards. The movement property that has the greatest effect on performance is learned faster and the credit assignment problem is solved efficiently.</p>
</sec>
</body>
<back><ref-list><title>References</title>
<ref id="pone.0055352-Kaelbling1"><label>1</label>
<mixed-citation publication-type="journal"><name><surname>Kaelbling</surname>
<given-names>LP</given-names>
</name>
, <name><surname>Littman</surname>
<given-names>ML</given-names>
</name>
, <name><surname>Moore</surname>
<given-names>AW</given-names>
</name>
(<year>1996</year>
) <article-title>Reinforcement learning: A Survey</article-title>
. <source>Journal of Artificial Intelligence Research</source>
<volume>4</volume>
: <fpage>237</fpage>
–<lpage>287</lpage>
</mixed-citation>
</ref>
<ref id="pone.0055352-Houk1"><label>2</label>
<mixed-citation publication-type="journal"><name><surname>Houk</surname>
<given-names>JC</given-names>
</name>
, <name><surname>Buckingham</surname>
<given-names>J</given-names>
</name>
, <name><surname>Barto</surname>
<given-names>AG</given-names>
</name>
(<year>1996</year>
) <article-title>Models of the cerebellum and motor learning</article-title>
. <source>Behavioral and Brain Sciences</source>
<volume>19</volume>
: <fpage>368</fpage>
–<lpage>379</lpage>
</mixed-citation>
</ref>
<ref id="pone.0055352-Berniker1"><label>3</label>
<mixed-citation publication-type="journal"><name><surname>Berniker</surname>
<given-names>M</given-names>
</name>
, <name><surname>Körding</surname>
<given-names>K</given-names>
</name>
(<year>2008</year>
) <article-title>Estimating the sources of motor errors for adaptation and generalization</article-title>
. <source>Nature Neuroscience</source>
<volume>11</volume>
: <fpage>1454</fpage>
–<lpage>1461</lpage>
</mixed-citation>
</ref>
<ref id="pone.0055352-Schultz1"><label>4</label>
<mixed-citation publication-type="journal"><name><surname>Schultz</surname>
<given-names>W</given-names>
</name>
(<year>2006</year>
) <article-title>Behavioral theories and the neurophysiology of reward</article-title>
. <source>Annual Review of Psychology</source>
<volume>57</volume>
: <fpage>87</fpage>
–<lpage>115</lpage>
</mixed-citation>
</ref>
<ref id="pone.0055352-Bernstein1"><label>5</label>
<mixed-citation publication-type="other">Bernstein NA (1967) The co-ordination and regulation of movements.</mixed-citation>
</ref>
<ref id="pone.0055352-Todorov1"><label>6</label>
<mixed-citation publication-type="journal"><name><surname>Todorov</surname>
<given-names>E</given-names>
</name>
, <name><surname>Jordan</surname>
<given-names>MI</given-names>
</name>
(<year>2002</year>
) <article-title>Optimal feedback control as a theory of motor coordination</article-title>
. <source>Nat Neurosci</source>
<volume>5</volume>
: <fpage>1226</fpage>
–<lpage>1235</lpage>
<pub-id pub-id-type="pmid">12404008</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Wolpert1"><label>7</label>
<mixed-citation publication-type="journal"><name><surname>Wolpert</surname>
<given-names>DM</given-names>
</name>
, <name><surname>Ghahramani</surname>
<given-names>Z</given-names>
</name>
, <name><surname>Flanagan</surname>
<given-names>JR</given-names>
</name>
(<year>2001</year>
) <article-title>Perspectives and problems in motor learning</article-title>
. <source>Trends in Cognitive Sciences</source>
<volume>5</volume>
: <fpage>487</fpage>
–<lpage>494</lpage>
<pub-id pub-id-type="pmid">11684481</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Scholz1"><label>8</label>
<mixed-citation publication-type="journal"><name><surname>Scholz</surname>
<given-names>JP</given-names>
</name>
, <name><surname>Schoner</surname>
<given-names>G</given-names>
</name>
, <name><surname>Latash</surname>
<given-names>ML</given-names>
</name>
(<year>2000</year>
) <article-title>Identifying the control structure of multijoint coordination during pistol shooting</article-title>
. <source>Experimental Brain Research</source>
<volume>135</volume>
: <fpage>382</fpage>
–<lpage>404</lpage>
</mixed-citation>
</ref>
<ref id="pone.0055352-Mller1"><label>9</label>
<mixed-citation publication-type="journal"><name><surname>Müller</surname>
<given-names>H</given-names>
</name>
, <name><surname>Sternad</surname>
<given-names>D</given-names>
</name>
(<year>2004</year>
) <article-title>Decomposition of variability in the execution of goal-oriented tasks: three components of skill improvement</article-title>
. <source>Journal of Experimental Psychology: Human Perception and Performance</source>
<volume>30</volume>
: <fpage>212</fpage>
–<lpage>233</lpage>
<pub-id pub-id-type="pmid">14769078</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Pennington1"><label>10</label>
<mixed-citation publication-type="journal"><name><surname>Pennington</surname>
<given-names>N</given-names>
</name>
, <name><surname>Hastie</surname>
<given-names>R</given-names>
</name>
(<year>1988</year>
) <article-title>Explanation-based decision making: effects of memory structure on judgment</article-title>
. <source>Journal of Experimental Psychology</source>
<volume>14</volume>
: <fpage>521</fpage>
–<lpage>533</lpage>
</mixed-citation>
</ref>
<ref id="pone.0055352-Kaelbling2"><label>11</label>
<mixed-citation publication-type="other">Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: A survey. arXiv preprint cs/9605103.</mixed-citation>
</ref>
<ref id="pone.0055352-Latash1"><label>12</label>
<mixed-citation publication-type="journal"><name><surname>Latash</surname>
<given-names>ML</given-names>
</name>
, <name><surname>Scholz</surname>
<given-names>JP</given-names>
</name>
, <name><surname>Sch ner</surname>
<given-names>G</given-names>
</name>
(<year>2002</year>
) <article-title>Motor control strategies revealed in the structure of motor variability</article-title>
. <source>Exercise and sport sciences reviews</source>
<volume>30</volume>
: <fpage>26</fpage>
<pub-id pub-id-type="pmid">11800496</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Scholz2"><label>13</label>
<mixed-citation publication-type="journal"><name><surname>Scholz</surname>
<given-names>JP</given-names>
</name>
, <name><surname>Schoner</surname>
<given-names>G</given-names>
</name>
(<year>1999</year>
) <article-title>The uncontrolled manifold concept: identifying control variables for a functional task</article-title>
. <source>Experimental Brain Research</source>
<volume>126</volume>
: <fpage>289</fpage>
–<lpage>306</lpage>
</mixed-citation>
</ref>
<ref id="pone.0055352-Mller2"><label>14</label>
<mixed-citation publication-type="journal"><name><surname>Müller</surname>
<given-names>H</given-names>
</name>
, <name><surname>Sternad</surname>
<given-names>D</given-names>
</name>
(<year>2003</year>
) <article-title>A randomization method for the calculation of covariation in multiple nonlinear relations: illustrated with the example of goal-directed movements</article-title>
. <source>Biological Cybernetics</source>
<volume>89</volume>
: <fpage>22</fpage>
–<lpage>33</lpage>
<pub-id pub-id-type="pmid">12836030</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Sternad1"><label>15</label>
<mixed-citation publication-type="journal"><name><surname>Sternad</surname>
<given-names>D</given-names>
</name>
, <name><surname>Abe</surname>
<given-names>MO</given-names>
</name>
, <name><surname>Hu</surname>
<given-names>X</given-names>
</name>
, <name><surname>Müller</surname>
<given-names>H</given-names>
</name>
(<year>2011</year>
) <article-title>Neuromotor Noise, Error Tolerance and Velocity-Dependent Costs in Skilled Performance</article-title>
. <source>PLoS computational biology</source>
<volume>7</volume>
: <fpage>e1002159</fpage>
<pub-id pub-id-type="pmid">21966262</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Martin1"><label>16</label>
<mixed-citation publication-type="journal"><name><surname>Martin</surname>
<given-names>TA</given-names>
</name>
, <name><surname>Norris</surname>
<given-names>SA</given-names>
</name>
, <name><surname>Greger</surname>
<given-names>BE</given-names>
</name>
, <name><surname>Thach</surname>
<given-names>WT</given-names>
</name>
(<year>2002</year>
) <article-title>Dynamic coordination of body parts during prism adaptation</article-title>
. <source>Journal of Neurophysiology</source>
<volume>88</volume>
: <fpage>1685</fpage>
<pub-id pub-id-type="pmid">12364498</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Kudo1"><label>17</label>
<mixed-citation publication-type="journal"><name><surname>Kudo</surname>
<given-names>K</given-names>
</name>
, <name><surname>Tsutsui</surname>
<given-names>S</given-names>
</name>
, <name><surname>Ishikura</surname>
<given-names>T</given-names>
</name>
, <name><surname>Ito</surname>
<given-names>T</given-names>
</name>
, <name><surname>Yamamoto</surname>
<given-names>Y</given-names>
</name>
(<year>2000</year>
) <article-title>Compensatory coordination of release parameters in a throwing task</article-title>
. <source>Journal of Motor Behavior</source>
<volume>32</volume>
: <fpage>337</fpage>
–<lpage>345</lpage>
<pub-id pub-id-type="pmid">11114227</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Latash2"><label>18</label>
<mixed-citation publication-type="journal"><name><surname>Latash</surname>
<given-names>ML</given-names>
</name>
, <name><surname>Scholz</surname>
<given-names>JP</given-names>
</name>
, <name><surname>Sch ner</surname>
<given-names>G</given-names>
</name>
(<year>2007</year>
) <article-title>Toward a new theory of motor synergies</article-title>
. <source>Motor Control</source>
<volume>11</volume>
: <fpage>276</fpage>
–<lpage>308</lpage>
<pub-id pub-id-type="pmid">17715460</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Sutton1"><label>19</label>
<mixed-citation publication-type="other">Sutton RS, Barto AG (1998) Reinforcement Learning. Cambridge, MA: MIT Press.</mixed-citation>
</ref>
<ref id="pone.0055352-Tolman1"><label>20</label>
<mixed-citation publication-type="other">Tolman EC, Honzik CH (1930) Introduction and removal of reward, and maze performance in rats. University of California Publications in Psychology.</mixed-citation>
</ref>
<ref id="pone.0055352-Fu1"><label>21</label>
<mixed-citation publication-type="journal"><name><surname>Fu</surname>
<given-names>WT</given-names>
</name>
, <name><surname>Anderson</surname>
<given-names>JR</given-names>
</name>
(<year>2006</year>
) <article-title>From recurrent choice to skill learning: A reinforcement-learning model</article-title>
. <source>Journal of experimental psychology: General</source>
<volume>135</volume>
: <fpage>184</fpage>
<pub-id pub-id-type="pmid">16719650</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Fu2"><label>22</label>
<mixed-citation publication-type="journal"><name><surname>Fu</surname>
<given-names>WT</given-names>
</name>
, <name><surname>Anderson</surname>
<given-names>JR</given-names>
</name>
(<year>2008</year>
) <article-title>Solving the credit assignment problem: Explicit and implicit learning of action sequences with probabilistic outcomes</article-title>
. <source>Psychological Research</source>
<volume>72</volume>
: <fpage>321</fpage>
–<lpage>330</lpage>
<pub-id pub-id-type="pmid">17447083</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0055352-Gordon1"><label>23</label>
<mixed-citation publication-type="other">Gordon D, Schultz A, Grefenstette J, Ballas J, Perez M (1994) User’s guide to the Navigation and Collision Avoidance Task. Naval Research Lab DC.</mixed-citation>
</ref>
<ref id="pone.0055352-Gullapalli1"><label>24</label>
<mixed-citation publication-type="other">Gullapalli V (1992) Reinforcement learning and its application to control: Citeseer.</mixed-citation>
</ref>
<ref id="pone.0055352-Houk2"><label>25</label>
<mixed-citation publication-type="journal"><name><surname>Houk</surname>
<given-names>JC</given-names>
</name>
, <name><surname>Buckingham</surname>
<given-names>JT</given-names>
</name>
, <name><surname>Barto</surname>
<given-names>AG</given-names>
</name>
(<year>1996</year>
) <article-title>Models of the cerebellum and motor learning</article-title>
. <source>Behavioral and Brain Sciences</source>
<volume>19</volume>
: <fpage>368</fpage>
–<lpage>383</lpage>
</mixed-citation>
</ref>
<ref id="pone.0055352-Wei1"><label>26</label>
<mixed-citation publication-type="journal"><name><surname>Wei</surname>
<given-names>K</given-names>
</name>
, <name><surname>Kording</surname>
<given-names>K</given-names>
</name>
(<year>2009</year>
) <article-title>Relevance of error: what drives motor adaptation?</article-title>
<source>Journal of Neurophysiology</source>
<volume>101</volume>
: <fpage>655</fpage>
–<lpage>664</lpage>
<pub-id pub-id-type="pmid">19019979</pub-id>
</mixed-citation>
</ref>
</ref-list>
</back>
</pmc>
<affiliations><list><country><li>République populaire de Chine</li>
<li>États-Unis</li>
</country>
<region><li>Illinois</li>
<li>Ohio</li>
</region>
<settlement><li>Pékin</li>
</settlement>
<orgName><li>Université de Pékin</li>
</orgName>
</list>
<tree><country name="États-Unis"><region name="Ohio"><name sortKey="Dam, Gregory" sort="Dam, Gregory" uniqKey="Dam G" first="Gregory" last="Dam">Gregory Dam</name>
</region>
<name sortKey="Kording, Konrad" sort="Kording, Konrad" uniqKey="Kording K" first="Konrad" last="Kording">Konrad Kording</name>
</country>
<country name="République populaire de Chine"><noRegion><name sortKey="Wei, Kunlin" sort="Wei, Kunlin" uniqKey="Wei K" first="Kunlin" last="Wei">Kunlin Wei</name>
</noRegion>
</country>
</tree>
</affiliations>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/HapticV1/Data/Ncbi/Merge
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 002522 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Ncbi/Merge/biblio.hfd -nk 002522 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Ticri/CIDE |area= HapticV1 |flux= Ncbi |étape= Merge |type= RBID |clé= PMC:3568147 |texte= Credit Assignment during Movement Reinforcement Learning }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Ncbi/Merge/RBID.i -Sk "pubmed:23408972" \ | HfdSelect -Kh $EXPLOR_AREA/Data/Ncbi/Merge/biblio.hfd \ | NlmPubMed2Wicri -a HapticV1
![]() | This area was generated with Dilib version V0.6.23. | ![]() |