The Communicability of Graphical Alternatives to Tabular Displays of Statistical Simulation Studies
Identifieur interne : 000313 ( Pmc/Checkpoint ); précédent : 000312; suivant : 000314The Communicability of Graphical Alternatives to Tabular Displays of Statistical Simulation Studies
Auteurs : Alex R. Cook ; Shanice W. L. TeoSource :
- PLoS ONE ; 2011.
Abstract
Simulation studies are often used to assess the frequency properties and optimality of statistical methods. They are typically reported in tables, which may contain hundreds of figures to be contrasted over multiple dimensions. To assess the degree to which these tables are fit for purpose, we performed a randomised cross-over experiment in which statisticians were asked to extract information from (i) such a table sourced from the literature and (ii) a graphical adaptation designed by the authors, and were timed and assessed for accuracy. We developed hierarchical models accounting for differences between individuals of different experience levels (under- and post-graduate), within experience levels, and between different table-graph pairs. In our experiment, information could be extracted quicker and, for less experienced participants, more accurately from graphical presentations than tabular displays. We also performed a literature review to assess the prevalence of hard-to-interpret design features in tables of simulation studies in three popular statistics journals, finding that many are presented innumerately. We recommend simulation studies be presented in graphical form.
Url:
DOI: 10.1371/journal.pone.0027974
PubMed: 22132184
PubMed Central: 3223202
Affiliations:
Links toward previous steps (curation, corpus...)
Links to Exploration step
PMC:3223202Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">The Communicability of Graphical Alternatives to Tabular Displays of Statistical Simulation Studies</title>
<author><name sortKey="Cook, Alex R" sort="Cook, Alex R" uniqKey="Cook A" first="Alex R." last="Cook">Alex R. Cook</name>
<affiliation><nlm:aff id="aff1"><addr-line>Saw Swee Hock School of Public Health, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</nlm:aff>
<wicri:noCountry code="subfield">Republic of Singapore</wicri:noCountry>
</affiliation>
<affiliation><nlm:aff id="aff2"><addr-line>Department of Statistics and Applied Probability, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</nlm:aff>
<wicri:noCountry code="subfield">Republic of Singapore</wicri:noCountry>
</affiliation>
<affiliation><nlm:aff id="aff3"><addr-line>Program in Health Services and Systems Research, Duke-NUS Graduate Medical School Singapore, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</nlm:aff>
<wicri:noCountry code="subfield">Republic of Singapore</wicri:noCountry>
</affiliation>
</author>
<author><name sortKey="Teo, Shanice W L" sort="Teo, Shanice W L" uniqKey="Teo S" first="Shanice W. L." last="Teo">Shanice W. L. Teo</name>
<affiliation><nlm:aff id="aff2"><addr-line>Department of Statistics and Applied Probability, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</nlm:aff>
<wicri:noCountry code="subfield">Republic of Singapore</wicri:noCountry>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PMC</idno>
<idno type="pmid">22132184</idno>
<idno type="pmc">3223202</idno>
<idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3223202</idno>
<idno type="RBID">PMC:3223202</idno>
<idno type="doi">10.1371/journal.pone.0027974</idno>
<date when="2011">2011</date>
<idno type="wicri:Area/Pmc/Corpus">000803</idno>
<idno type="wicri:Area/Pmc/Curation">000803</idno>
<idno type="wicri:Area/Pmc/Checkpoint">000313</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">The Communicability of Graphical Alternatives to Tabular Displays of Statistical Simulation Studies</title>
<author><name sortKey="Cook, Alex R" sort="Cook, Alex R" uniqKey="Cook A" first="Alex R." last="Cook">Alex R. Cook</name>
<affiliation><nlm:aff id="aff1"><addr-line>Saw Swee Hock School of Public Health, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</nlm:aff>
<wicri:noCountry code="subfield">Republic of Singapore</wicri:noCountry>
</affiliation>
<affiliation><nlm:aff id="aff2"><addr-line>Department of Statistics and Applied Probability, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</nlm:aff>
<wicri:noCountry code="subfield">Republic of Singapore</wicri:noCountry>
</affiliation>
<affiliation><nlm:aff id="aff3"><addr-line>Program in Health Services and Systems Research, Duke-NUS Graduate Medical School Singapore, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</nlm:aff>
<wicri:noCountry code="subfield">Republic of Singapore</wicri:noCountry>
</affiliation>
</author>
<author><name sortKey="Teo, Shanice W L" sort="Teo, Shanice W L" uniqKey="Teo S" first="Shanice W. L." last="Teo">Shanice W. L. Teo</name>
<affiliation><nlm:aff id="aff2"><addr-line>Department of Statistics and Applied Probability, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</nlm:aff>
<wicri:noCountry code="subfield">Republic of Singapore</wicri:noCountry>
</affiliation>
</author>
</analytic>
<series><title level="j">PLoS ONE</title>
<idno type="e-ISSN">1932-6203</idno>
<imprint><date when="2011">2011</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en"><p>Simulation studies are often used to assess the frequency properties and optimality of statistical methods. They are typically reported in tables, which may contain hundreds of figures to be contrasted over multiple dimensions. To assess the degree to which these tables are fit for purpose, we performed a randomised cross-over experiment in which statisticians were asked to extract information from (i) such a table sourced from the literature and (ii) a graphical adaptation designed by the authors, and were timed and assessed for accuracy. We developed hierarchical models accounting for differences between individuals of different experience levels (under- and post-graduate), within experience levels, and between different table-graph pairs. In our experiment, information could be extracted quicker and, for less experienced participants, more accurately from graphical presentations than tabular displays. We also performed a literature review to assess the prevalence of hard-to-interpret design features in tables of simulation studies in three popular statistics journals, finding that many are presented innumerately. We recommend simulation studies be presented in graphical form.</p>
</div>
</front>
<back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Farquhar, Ab" uniqKey="Farquhar A">AB Farquhar</name>
</author>
<author><name sortKey="Farquhar, H" uniqKey="Farquhar H">H Farquhar</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wainer, H" uniqKey="Wainer H">H Wainer</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Ehrenberg, Asc" uniqKey="Ehrenberg A">ASC Ehrenberg</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Mahon, Bh" uniqKey="Mahon B">BH Mahon</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Gelman, A" uniqKey="Gelman A">A Gelman</name>
</author>
<author><name sortKey="Pasarica, C" uniqKey="Pasarica C">C Pasarica</name>
</author>
<author><name sortKey="Dodhia, R" uniqKey="Dodhia R">R Dodhia</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Feinberg, Ra" uniqKey="Feinberg R">RA Feinberg</name>
</author>
<author><name sortKey="Wainer, H" uniqKey="Wainer H">H Wainer</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Gelman, A" uniqKey="Gelman A">A Gelman</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wainer, H" uniqKey="Wainer H">H Wainer</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Briggs, Wm" uniqKey="Briggs W">WM Briggs</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Friendly, M" uniqKey="Friendly M">M Friendly</name>
</author>
<author><name sortKey="Kwan, E" uniqKey="Kwan E">E Kwan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wills, G" uniqKey="Wills G">G Wills</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Gelman, A" uniqKey="Gelman A">A Gelman</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Remus, W" uniqKey="Remus W">W Remus</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Meyer, J" uniqKey="Meyer J">J Meyer</name>
</author>
<author><name sortKey="Shinar, D" uniqKey="Shinar D">D Shinar</name>
</author>
<author><name sortKey="Leiser, D" uniqKey="Leiser D">D Leiser</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Meyer, J" uniqKey="Meyer J">J Meyer</name>
</author>
<author><name sortKey="Shamo, Mk" uniqKey="Shamo M">MK Shamo</name>
</author>
<author><name sortKey="Gopher, D" uniqKey="Gopher D">D Gopher</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Spence, I" uniqKey="Spence I">I Spence</name>
</author>
<author><name sortKey="Lewandowsky, S" uniqKey="Lewandowsky S">S Lewandowsky</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Tufte, Er" uniqKey="Tufte E">ER Tufte</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Cleveland, Ws" uniqKey="Cleveland W">WS Cleveland</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Cleveland, Ws" uniqKey="Cleveland W">WS Cleveland</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Zhang, M" uniqKey="Zhang M">M Zhang</name>
</author>
<author><name sortKey="Zhang, D" uniqKey="Zhang D">D Zhang</name>
</author>
<author><name sortKey="Wells, Mt" uniqKey="Wells M">MT Wells</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Ho, Yhs" uniqKey="Ho Y">YHS Ho</name>
</author>
<author><name sortKey="Lee, Sms" uniqKey="Lee S">SMS Lee</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Conti, Pl" uniqKey="Conti P">PL Conti</name>
</author>
<author><name sortKey="De Giovanni, L" uniqKey="De Giovanni L">L De Giovanni</name>
</author>
<author><name sortKey="Stoev, Sa" uniqKey="Stoev S">SA Stoev</name>
</author>
<author><name sortKey="Taqqu, Ms" uniqKey="Taqqu M">MS Taqqu</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Loh, Wy" uniqKey="Loh W">WY Loh</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Roy, A" uniqKey="Roy A">A Roy</name>
</author>
<author><name sortKey="Falk, B" uniqKey="Falk B">B Falk</name>
</author>
<author><name sortKey="Fuller, Wa" uniqKey="Fuller W">WA Fuller</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wang, C" uniqKey="Wang C">C Wang</name>
</author>
<author><name sortKey="Paik, Mc" uniqKey="Paik M">MC Paik</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Murrell, P" uniqKey="Murrell P">P Murrell</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Gelman, A" uniqKey="Gelman A">A Gelman</name>
</author>
<author><name sortKey="Hill, J" uniqKey="Hill J">J Hill</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Metropolis, N" uniqKey="Metropolis N">N Metropolis</name>
</author>
<author><name sortKey="Rosenbluth, Aw" uniqKey="Rosenbluth A">AW Rosenbluth</name>
</author>
<author><name sortKey="Rosenbluth, Mn" uniqKey="Rosenbluth M">MN Rosenbluth</name>
</author>
<author><name sortKey="Teller, Ah" uniqKey="Teller A">AH Teller</name>
</author>
<author><name sortKey="Teller, E" uniqKey="Teller E">E Teller</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Hastings, Wk" uniqKey="Hastings W">WK Hastings</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Gelman, A" uniqKey="Gelman A">A Gelman</name>
</author>
<author><name sortKey="Carlin, Jb" uniqKey="Carlin J">JB Carlin</name>
</author>
<author><name sortKey="Stern, Hs" uniqKey="Stern H">HS Stern</name>
</author>
<author><name sortKey="Rubin, Db" uniqKey="Rubin D">DB Rubin</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Gilks, Wr" uniqKey="Gilks W">WR Gilks</name>
</author>
<author><name sortKey="Richardson, S" uniqKey="Richardson S">S Richardson</name>
</author>
<author><name sortKey="Spiegelhalter, D" uniqKey="Spiegelhalter D">D Spiegelhalter</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Thomas, A" uniqKey="Thomas A">A Thomas</name>
</author>
<author><name sortKey="O Hara, B" uniqKey="O Hara B">B O'Hara</name>
</author>
<author><name sortKey="Ligges, U" uniqKey="Ligges U">U Ligges</name>
</author>
<author><name sortKey="Sturtz, S" uniqKey="Sturtz S">S Sturtz</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Gelman, A" uniqKey="Gelman A">A Gelman</name>
</author>
<author><name sortKey="Rubin, Db" uniqKey="Rubin D">DB Rubin</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Brooks, Sp" uniqKey="Brooks S">SP Brooks</name>
</author>
<author><name sortKey="Gelman, A" uniqKey="Gelman A">A Gelman</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Plummer, M" uniqKey="Plummer M">M Plummer</name>
</author>
<author><name sortKey="Best, N" uniqKey="Best N">N Best</name>
</author>
<author><name sortKey="Cowles, K" uniqKey="Cowles K">K Cowles</name>
</author>
<author><name sortKey="Vines, K" uniqKey="Vines K">K Vines</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wainer, H" uniqKey="Wainer H">H Wainer</name>
</author>
<author><name sortKey="Lysen, S" uniqKey="Lysen S">S Lysen</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wainer, H" uniqKey="Wainer H">H Wainer</name>
</author>
<author><name sortKey="Larsen, M" uniqKey="Larsen M">M Larsen</name>
</author>
</analytic>
</biblStruct>
</listBibl>
</div1>
</back>
</TEI>
<pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">PLoS One</journal-id>
<journal-id journal-id-type="publisher-id">plos</journal-id>
<journal-id journal-id-type="pmc">plosone</journal-id>
<journal-title-group><journal-title>PLoS ONE</journal-title>
</journal-title-group>
<issn pub-type="epub">1932-6203</issn>
<publisher><publisher-name>Public Library of Science</publisher-name>
<publisher-loc>San Francisco, USA</publisher-loc>
</publisher>
</journal-meta>
<article-meta><article-id pub-id-type="pmid">22132184</article-id>
<article-id pub-id-type="pmc">3223202</article-id>
<article-id pub-id-type="publisher-id">PONE-D-11-13842</article-id>
<article-id pub-id-type="doi">10.1371/journal.pone.0027974</article-id>
<article-categories><subj-group subj-group-type="heading"><subject>Research Article</subject>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Computer Science</subject>
<subj-group><subject>Computerized Simulations</subject>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Mathematics</subject>
<subj-group><subject>Statistics</subject>
<subj-group><subject>Statistical Methods</subject>
</subj-group>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Medicine</subject>
<subj-group><subject>Clinical Research Design</subject>
<subj-group><subject>Statistical Methods</subject>
</subj-group>
</subj-group>
<subj-group><subject>Mental Health</subject>
<subj-group><subject>Psychology</subject>
<subj-group><subject>Cognitive Psychology</subject>
<subj-group><subject>Learning</subject>
</subj-group>
</subj-group>
</subj-group>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Social and Behavioral Sciences</subject>
<subj-group><subject>Psychology</subject>
<subj-group><subject>Cognitive Psychology</subject>
<subj-group><subject>Learning</subject>
</subj-group>
</subj-group>
</subj-group>
</subj-group>
</article-categories>
<title-group><article-title>The Communicability of Graphical Alternatives to Tabular Displays of Statistical Simulation Studies</article-title>
<alt-title alt-title-type="running-head">Graphs Versus Tables for Simulation Studies</alt-title>
</title-group>
<contrib-group><contrib contrib-type="author"><name><surname>Cook</surname>
<given-names>Alex R.</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2"><sup>2</sup>
</xref>
<xref ref-type="aff" rid="aff3"><sup>3</sup>
</xref>
<xref ref-type="corresp" rid="cor1"><sup>*</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Teo</surname>
<given-names>Shanice W. L.</given-names>
</name>
<xref ref-type="aff" rid="aff2"><sup>2</sup>
</xref>
</contrib>
</contrib-group>
<aff id="aff1"><label>1</label>
<addr-line>Saw Swee Hock School of Public Health, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</aff>
<aff id="aff2"><label>2</label>
<addr-line>Department of Statistics and Applied Probability, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</aff>
<aff id="aff3"><label>3</label>
<addr-line>Program in Health Services and Systems Research, Duke-NUS Graduate Medical School Singapore, National University of Singapore, Singapore, Republic of Singapore</addr-line>
</aff>
<contrib-group><contrib contrib-type="editor"><name><surname>Biondi-Zoccai</surname>
<given-names>Giuseppe</given-names>
</name>
<role>Editor</role>
<xref ref-type="aff" rid="edit1"></xref>
</contrib>
</contrib-group>
<aff id="edit1">University of Modena and Reggio Emilia, Italy</aff>
<author-notes><corresp id="cor1">* E-mail: <email>alex.richard.cook@gmail.com</email>
</corresp>
<fn fn-type="con"><p>Conceived and designed the experiments: ARC SWLT. Performed the experiments: SWLT. Analyzed the data: ARC SWLT. Wrote the paper: ARC SWLT.</p>
</fn>
</author-notes>
<pub-date pub-type="collection"><year>2011</year>
</pub-date>
<pub-date pub-type="epub"><day>23</day>
<month>11</month>
<year>2011</year>
</pub-date>
<volume>6</volume>
<issue>11</issue>
<elocation-id>e27974</elocation-id>
<history><date date-type="received"><day>20</day>
<month>7</month>
<year>2011</year>
</date>
<date date-type="accepted"><day>28</day>
<month>10</month>
<year>2011</year>
</date>
</history>
<permissions><copyright-statement>Cook, Teo. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.</copyright-statement>
<copyright-year>2011</copyright-year>
</permissions>
<abstract><p>Simulation studies are often used to assess the frequency properties and optimality of statistical methods. They are typically reported in tables, which may contain hundreds of figures to be contrasted over multiple dimensions. To assess the degree to which these tables are fit for purpose, we performed a randomised cross-over experiment in which statisticians were asked to extract information from (i) such a table sourced from the literature and (ii) a graphical adaptation designed by the authors, and were timed and assessed for accuracy. We developed hierarchical models accounting for differences between individuals of different experience levels (under- and post-graduate), within experience levels, and between different table-graph pairs. In our experiment, information could be extracted quicker and, for less experienced participants, more accurately from graphical presentations than tabular displays. We also performed a literature review to assess the prevalence of hard-to-interpret design features in tables of simulation studies in three popular statistics journals, finding that many are presented innumerately. We recommend simulation studies be presented in graphical form.</p>
</abstract>
<counts><page-count count="7"></page-count>
</counts>
</article-meta>
</front>
<body><sec id="s1"><title>Introduction</title>
<p>Simulation studies are vital in assessing statistical methods. They allow frequency properties or other criteria to be compared for competing methods, thus providing evidence to facilitate the establishment of new methods and, subsequently, their potential application to problems in science, medicine and the social sciences. In evaluating methodologies, simulation studies: (i) provide a cost-effective way to quantify potential performance for a large range of scenarios, spanning different combinations of sample sizes and underlying parameters, (ii) allow average performance to be estimated under repeat Monte Carlo sampling and (iii) facilitate comparison of estimates against the “true” system underlying the simulations, none of which is really achievable via genuine applications, as gratifying as those are.</p>
<p>As a consequence, many statistics journals brim with simulation studies. A challenge to the authors of such papers is deciding how to present simulated results. Typically, at least two methods are contrasted with respect to several objective functions, over a design space of at least one dimension (and often several); it is this complexity that confounds simple presentation. The usual approach is to present the simulation study in one or more tables.</p>
<p>It has long been pointed out that tabular displays of data (and by extension simulated data) can be difficult to interpret if constructed badly, with the task of taking information from a table having famously been likened to that of <italic>extracting sunlight from a cucumber</italic>
<xref ref-type="bibr" rid="pone.0027974-Farquhar1">[1]</xref>
(as cited in <xref ref-type="bibr" rid="pone.0027974-Wainer1">[2]</xref>
). Ehrenberg <xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
and discussants of his and Mahon's <xref ref-type="bibr" rid="pone.0027974-Mahon1">[4]</xref>
papers note “commonsense” advice going back to the 1910s on how to present tabular information. Ehrenberg <xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
and, later, Wainer <xref ref-type="bibr" rid="pone.0027974-Wainer1">[2]</xref>
outline a series of guidelines to present quantitative information numerately: round heavily (they recommend to two effective digits), provide anchoring via averages, make primary comparisons vertically, use effective ordering and good use of space. Ehrenberg <xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
argues convincingly that when a reader finds a table of figures hard to understand, even when told what to look for (his “weak criterion” for a good table), the fault lies with the producer of the table. As in good writing, the onus is on the author to facilitate the reader's comprehension. Our experience has been that many of the guidelines presented by Ehrenberg <xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
and Wainer <xref ref-type="bibr" rid="pone.0027974-Wainer1">[2]</xref>
are frequently broken by statisticians when presenting their own work in papers, seminars or conferences, and that many journals adopt tabular formatting requirements that do not abide by these guidelines.</p>
<p>In presenting simulation studies, authors have a choice of media: text, table or graph. Graphs have been recommended to display relationships or comparisons, tables for values <xref ref-type="bibr" rid="pone.0027974-Mahon1">[4]</xref>
, <xref ref-type="bibr" rid="pone.0027974-Gelman1">[5]</xref>
, but a review of an issue of the <italic>Journal of the American Statistical Association</italic>
suggested that most tables presented therein were used for comparisons <xref ref-type="bibr" rid="pone.0027974-Gelman1">[5]</xref>
, a task to which they are suboptimal. In their aforementioned review, Gelman et al. <xref ref-type="bibr" rid="pone.0027974-Gelman1">[5]</xref>
demonstrated that it is possible to create graphical variants of tables that are as compact as the table they might replace for a range of examples. A recent review found that, even in the <italic>Journal of Computational and</italic>
Graphical <italic>Statistics</italic>
, tables accounted for a third of all displays over the period 2005 to 2010 <xref ref-type="bibr" rid="pone.0027974-Feinberg1">[6]</xref>
.</p>
<p>Several explanations for the preference for tables over graphs have recently been suggested <xref ref-type="bibr" rid="pone.0027974-Gelman2">[7]</xref>
–<xref ref-type="bibr" rid="pone.0027974-Gelman3">[12]</xref>
. We believe that one further reason why the adoption of graphical displays of simulation studies is non-universal is the relative shortage of quantitative evidence in favour of graphs (or, contrariwise, tables), especially for constructions as complex as in multidimensional simulation studies. Previous research has been carried out primarily on small displays (e.g. refs. <xref ref-type="bibr" rid="pone.0027974-Remus1">[13]</xref>
–<xref ref-type="bibr" rid="pone.0027974-Spence1">[16]</xref>
) and as a consequence it is unclear how relevant previous findings (which have been mixed, with some supporting graphs and others tables, see the summary in ref. <xref ref-type="bibr" rid="pone.0027974-Gelman1">[5]</xref>
) are to the problem of displaying large simulation study results. In addition, most have focused on the display of actual data, whereas the numerical results of simulation studies typically consist of more abstract performance metrics to which, being less grounded in our everyday experiences, it may be harder to relate. At the same time, most previous studies have generally abided by the rules of good tabular design (though not always of good graphical design, as espoused by Tufte <xref ref-type="bibr" rid="pone.0027974-Tufte1">[17]</xref>
and Cleveland <xref ref-type="bibr" rid="pone.0027974-Cleveland1">[18]</xref>
, <xref ref-type="bibr" rid="pone.0027974-Cleveland2">[19]</xref>
<italic>inter alia</italic>
) and therefore are aspirational rather than descriptive of the current state of numerical information in methodological statistics.</p>
<p>To assess the relative communicability of tabular and graphical displays of large simulation studies, we designed and performed a randomised cross-over experiment to compare quantitatively the speed and accuracy at which information could be extracted from graphical or tabular displays by statisticians with varying levels of experience in statistical research. We analysed the results using an hierarchical model accounting for the heterogeneity in intrinsic difficulty between a set of representative tables selected from the literature and matching graphs designed by the authors, and between individual differences in speed and accuracy. This we fitted in the Bayesian paradigm. We also performed a systematic review of three statistical journals–<italic>the Journal of the American Statistical Association</italic>
, <italic>the Annals of Statistics</italic>
and <italic>Statistica Sinica</italic>
–to assess the prevalence of bad design features in tabular displays of simulation studies.</p>
</sec>
<sec sec-type="materials|methods" id="s2"><title>Materials and Methods</title>
<sec id="s2a"><title>Ethics statement</title>
<p>Participants in the experiment provided written informed consent and were able to withdraw from the study at any point before the anonymisation of their data. The study design was approved by the Institutional Review Board of the National University of Singapore.</p>
</sec>
<sec id="s2b"><title>Review of literature</title>
<p>We investigated the prevalence of bad design features in tabular displays of simulation studies via a systematic survey of three statistical journals: two of the most prestigious, the <italic>Annals of Statistics</italic>
and the <italic>Journal of the American Statistical Association</italic>
, and another, <italic>Statistica Sinica</italic>
, which is well regarded. We limited attention to tables portraying simulation studies, and performed a census of all such tables from all issues of these three journals published in the year 2009. We chose five criteria with reference to the discussion by Ehrenberg <xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
and Wainer <xref ref-type="bibr" rid="pone.0027974-Wainer1">[2]</xref>
as a means of assessing how well constructed these tables were, these being</p>
<list list-type="bullet"><list-item><p>the number of entries (the fewer, the easier to comprehend),</p>
</list-item>
<list-item><p>the maximum number of (non-leading zero) digits presented (ditto),</p>
</list-item>
<list-item><p>binary indicators coding the alignment of decimal places (aligned is preferred),</p>
</list-item>
<list-item><p>vertical versus horizontal comparisons (vertical is preferred), and</p>
</list-item>
<list-item><p>the presence of parenthesis (often [unnecessary] visual clutter).</p>
</list-item>
</list>
</sec>
<sec id="s2c"><title>Experimental protocol</title>
<p>Twenty volunteers participated in the randomised cross-over experiment. Ten were undergraduate students near the end of a BSc in statistics at the National University of Singapore (NUS), with some (limited) exposure to statistical methodological research. Another ten were either current postgraduate research students pursuing a PhD in statistics or were faculty members with research interests in statistics or applied probability and holding a PhD in statistics or an allied discipline, again at NUS. The dichotomisation into two groups was to allow us to assess the notion that statistical research experience might influence the perceived relative difficulty between tabular and graphical displays.</p>
<p>We selected six tables of simulation studies from the literature (refs. <xref ref-type="bibr" rid="pone.0027974-Zhang1">[20]</xref>
–<xref ref-type="bibr" rid="pone.0027974-Wang1">[25]</xref>
) to be one half of the experimental stimuli. Our intention was that these would cover the most common types of simulation study displays, and to this end they were selected purposefully, not randomly. The tables selected and a quantitative summary of their complexity are presented in <xref ref-type="table" rid="pone-0027974-t001">Table 1</xref>
.</p>
<table-wrap id="pone-0027974-t001" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0027974.t001</object-id>
<label>Table 1</label>
<caption><title>Details of the six tables taken from the literature used as stimuli in the cross-over experiment.</title>
</caption>
<alternatives><graphic id="pone-0027974-t001-1" xlink:href="pone.0027974.t001"></graphic>
<table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
</colgroup>
<thead><tr><td align="left" rowspan="1" colspan="1">Source table</td>
<td align="left" rowspan="1" colspan="1">Source ref.</td>
<td align="left" rowspan="1" colspan="1">Precision</td>
<td align="left" rowspan="1" colspan="1">Number of methods</td>
<td align="left" rowspan="1" colspan="1">Comparison dimensionality</td>
</tr>
</thead>
<tbody><tr><td align="left" rowspan="1" colspan="1">5.1</td>
<td align="left" rowspan="1" colspan="1"><xref ref-type="bibr" rid="pone.0027974-Zhang1">[20]</xref>
</td>
<td align="left" rowspan="1" colspan="1">4</td>
<td align="left" rowspan="1" colspan="1">6</td>
<td align="left" rowspan="1" colspan="1">3</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">4</td>
<td align="left" rowspan="1" colspan="1"><xref ref-type="bibr" rid="pone.0027974-Ho1">[21]</xref>
</td>
<td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1">3</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1"><xref ref-type="bibr" rid="pone.0027974-Conti1">[22]</xref>
</td>
<td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1">3</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">1</td>
<td align="left" rowspan="1" colspan="1"><xref ref-type="bibr" rid="pone.0027974-Loh1">[23]</xref>
</td>
<td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1">5</td>
<td align="left" rowspan="1" colspan="1">2</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1"><xref ref-type="bibr" rid="pone.0027974-Roy1">[24]</xref>
</td>
<td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1">5</td>
<td align="left" rowspan="1" colspan="1">1</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">1</td>
<td align="left" rowspan="1" colspan="1"><xref ref-type="bibr" rid="pone.0027974-Wang1">[25]</xref>
</td>
<td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1">11</td>
<td align="left" rowspan="1" colspan="1">4</td>
</tr>
</tbody>
</table>
</alternatives>
<table-wrap-foot><fn id="nt101"><label></label>
<p>The other stimuli–graphical adaptations of the tables below, designed by the authors– appear in <xref ref-type="supplementary-material" rid="pone.0027974.s001">Supporting Information S1</xref>
. The maximum precision in non-leading zero digits, the number of methods being compared, and the number of other dimensions (e.g. sample size, or effect size) along which comparisons could be made, are tabulated.</p>
</fn>
</table-wrap-foot>
</table-wrap>
<p>For each of these six tables we created a matching graph, using the grid package <xref ref-type="bibr" rid="pone.0027974-Murrell1">[26]</xref>
within the R statistical environment <xref ref-type="bibr" rid="pone.0027974-R1">[27]</xref>
. The package grid allows the user the power to design graphs to the minutest detail, and probably attains the current apex of statistical graphics. We used grid to maximise the information content of the six matching graphs by controlling spacing, axis labelling and colouring of graph elements. The resulting graphs are presented in <xref ref-type="supplementary-material" rid="pone.0027974.s001">Supporting Information S1</xref>
.</p>
<p>For all six table/graph pairs, we then wrote a set of five questions which participants were to answer as quickly and accurately as possible. The questions (provided in <xref ref-type="supplementary-material" rid="pone.0027974.s001">Supporting Information S1</xref>
) were written to emulate the sort of questions the authors ask when looking at tables of simulation studies, for instance: <italic>which method has the least accurate estimated coverage probability in all cases considered?</italic>
and <italic>which test usually has the lowest power?</italic>
Many of the questions were rather difficult to answer since they required comparing numbers across several dimensions at one time–a common challenge in trying to understand simulation studies. All questions were constructed to allow an answer that could be marked correct or not, although this constraint may have lessened the realism of the experiment.</p>
<p>The experimental protocol was as follows. Each participant was allocated in advance one graph and one table, drawn uniformly randomly from all 30 pairs of non-matching graphs and tables using a pseudo-random number generator. Participants sat in a quiet room with one or both of the authors. After providing written informed consent, they were presented with one stimulus, either a graph or a table (the ordering selected pseudo-randomly <italic>a priori</italic>
), and some background information on the nature of the research problem tackled in the paper containing the original table. They were given as long as they wished to read this. They were then presented with the associated list of five questions for that stimulus and the time taken to answer each question recorded. After completing the five questions, the process was repeated using the other stimulus. Afterwards, the authors assigned a binary correctness score to each answer using a pre-designed rubric. The ten times in seconds and scores were then entered anonymously into a database along with the experience level of the participant (under- or post-graduate) and a categorical indicator of the stimuli used. The data are provided in <xref ref-type="supplementary-material" rid="pone.0027974.s003">Supporting Information S2</xref>
and <xref ref-type="supplementary-material" rid="pone.0027974.s003">S3</xref>
.</p>
<p>No participants elected to withdraw from the study.</p>
</sec>
<sec id="s2d"><title>Data analysis</title>
<p>To account for the structured nature of the experiment–repeat observations from different combinations of graphs/tables and study participants of two different experience levels–we use multilevel models (see <xref ref-type="bibr" rid="pone.0027974-Gelman4">[28]</xref>
for example) fit in the Bayesian paradigm for both timing and accuracy data. The Bayesian approach is attractive here for several reasons: (i) the model has to have multiple levels to represent the structure of the data, and Bayesian methods are particularly suited to such models; (ii) some elements of the data hierarchy have small samples–ten individuals per experience level, six stimuli–and hence any assumption of having reached asymptotic normality has to be suspect; and (iii) a Bayesian approach readily allows estimates and uncertainty intervals of functions of the parameters to be obtained. For the timing data, we work with the natural logarithm of the time in seconds, and take the mean log-time for the <inline-formula><inline-graphic xlink:href="pone.0027974.e001.jpg" mimetype="image"></inline-graphic>
</inline-formula>
th measurement for the <inline-formula><inline-graphic xlink:href="pone.0027974.e002.jpg" mimetype="image"></inline-graphic>
</inline-formula>
th treatment on the <inline-formula><inline-graphic xlink:href="pone.0027974.e003.jpg" mimetype="image"></inline-graphic>
</inline-formula>
th individual (<inline-formula><inline-graphic xlink:href="pone.0027974.e004.jpg" mimetype="image"></inline-graphic>
</inline-formula>
, <inline-formula><inline-graphic xlink:href="pone.0027974.e005.jpg" mimetype="image"></inline-graphic>
</inline-formula>
, <inline-formula><inline-graphic xlink:href="pone.0027974.e006.jpg" mimetype="image"></inline-graphic>
</inline-formula>
) to be: <disp-formula><graphic xlink:href="pone.0027974.e007"></graphic>
<label>(1)</label>
</disp-formula>
</p>
<p>where</p>
<list list-type="bullet"><list-item><p><inline-formula><inline-graphic xlink:href="pone.0027974.e008.jpg" mimetype="image"></inline-graphic>
</inline-formula>
captures the speed of individual <inline-formula><inline-graphic xlink:href="pone.0027974.e009.jpg" mimetype="image"></inline-graphic>
</inline-formula>
and is assumed to come from a normal distribution with standard deviation <inline-formula><inline-graphic xlink:href="pone.0027974.e010.jpg" mimetype="image"></inline-graphic>
</inline-formula>
and, for undergraduate participants, mean zero, while for postgraduate participants, the mean was the free parameter, <inline-formula><inline-graphic xlink:href="pone.0027974.e011.jpg" mimetype="image"></inline-graphic>
</inline-formula>
;</p>
</list-item>
<list-item><p><inline-formula><inline-graphic xlink:href="pone.0027974.e012.jpg" mimetype="image"></inline-graphic>
</inline-formula>
quantifies the difficulty of the table or graph <inline-formula><inline-graphic xlink:href="pone.0027974.e013.jpg" mimetype="image"></inline-graphic>
</inline-formula>
selected randomly from the bank for which <inline-formula><inline-graphic xlink:href="pone.0027974.e014.jpg" mimetype="image"></inline-graphic>
</inline-formula>
is answering questions, and is assumed to come from a mean zero normal distribution with standard deviation <inline-formula><inline-graphic xlink:href="pone.0027974.e015.jpg" mimetype="image"></inline-graphic>
</inline-formula>
;</p>
</list-item>
<list-item><p><inline-formula><inline-graphic xlink:href="pone.0027974.e016.jpg" mimetype="image"></inline-graphic>
</inline-formula>
captures the effect on speed of being presented a table (for <inline-formula><inline-graphic xlink:href="pone.0027974.e017.jpg" mimetype="image"></inline-graphic>
</inline-formula>
) rather than a graph (<inline-formula><inline-graphic xlink:href="pone.0027974.e018.jpg" mimetype="image"></inline-graphic>
</inline-formula>
); allowing an interaction with the experience of individual <inline-formula><inline-graphic xlink:href="pone.0027974.e019.jpg" mimetype="image"></inline-graphic>
</inline-formula>
we have <inline-formula><inline-graphic xlink:href="pone.0027974.e020.jpg" mimetype="image"></inline-graphic>
</inline-formula>
fixed to 0, <inline-formula><inline-graphic xlink:href="pone.0027974.e021.jpg" mimetype="image"></inline-graphic>
</inline-formula>
for undergraduates, and <inline-formula><inline-graphic xlink:href="pone.0027974.e022.jpg" mimetype="image"></inline-graphic>
</inline-formula>
for postgraduates, the latter two free parameters.</p>
</list-item>
</list>
<p>Note that some parameters described above are set to zero for the sake of identifiability. We assume homoskedasticity of the distribution of log-times. All parameters are assigned non-informative prior distributions (see <xref ref-type="supplementary-material" rid="pone.0027974.s001">Supporting Information S1</xref>
for details).</p>
<p>For the accuracy measurements, a very similar model was fitted, using a Bernoulli distribution for the classification of each answer as correct or not, with the logit of the probability of an accurate answer replacing the mean log-time in equation (1) and the assumption of homoskedasticity being dropped.</p>
<p>The models were fitted using Markov chain Monte Carlo sampling <xref ref-type="bibr" rid="pone.0027974-Metropolis1">[29]</xref>
–<xref ref-type="bibr" rid="pone.0027974-Gilks1">[32]</xref>
in OpenBUGS version 3.1.1 <xref ref-type="bibr" rid="pone.0027974-Thomas1">[33]</xref>
with four samplers with over-dispersed starting points and 100 000 iterations following a burn-in of 20 000 and every 10th iteration retained for later analysis. Convergence was assessed using the Gelman–Rubin <xref ref-type="bibr" rid="pone.0027974-Gelman6">[34]</xref>
and Brooks–Gelman <xref ref-type="bibr" rid="pone.0027974-Brooks1">[35]</xref>
diagnostics in the CODA package <xref ref-type="bibr" rid="pone.0027974-Plummer1">[36]</xref>
for R <xref ref-type="bibr" rid="pone.0027974-R1">[27]</xref>
, as well as by visual assessment of the resulting trace plots.</p>
<p>The retained output was exported to R via CODA for post-processing, involving converting back from the logarithmic or logistic scale to the original data scale to facilitate interpretation. Estimates are accompanied by 95% (credible) intervals (95%I).</p>
</sec>
</sec>
<sec id="s3"><title>Results</title>
<sec id="s3a"><title>Review of literature</title>
<p>From 313 articles published in the three focus journals in 2009, we identified 184 (59%) that contained some form of simulation study. A representation of the prevalence of undesirable design features is presented in <xref ref-type="fig" rid="pone-0027974-g001">Figure 1</xref>
. The focus journals for the most part succeed in avoiding visual problems that can be addressed at the type-setting stage, with most tables having decimal places aligned (91%), thereby enabling the reader to determine the order of magnitude of numbers at a glance, and avoiding the use of parenthesis (74%), which add unnecessary visual clutter.</p>
<fig id="pone-0027974-g001" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0027974.g001</object-id>
<label>Figure 1</label>
<caption><title>Prevalence of undesirable design features in tabular displays in statistics journals.</title>
<p>All tables containing results of simulation studies in articles published in the <italic>Journal of the American Statistical Association</italic>
(JASA), <italic>Annals of Statistics</italic>
(AS) and <italic>Statistica Sinica</italic>
(SS) during the year 2009 comprise the population reviewed. The proportion satisfying various criteria are marked: for the first three, the higher the proportion the better, while smaller tables with fewer significant figures (sig figs) are preferred. Overall proportions are indicated with a solid circle, within journal proportions by a hollow circle. For each criterion, if one journal did notably worse than the others, its proportion is labelled.</p>
</caption>
<graphic xlink:href="pone.0027974.g001"></graphic>
</fig>
<p>Several more inherent obstacles to interpretation remain. Some 38% of tables are set up so that the main direction along which comparisons are made is horizontal, rather than vertical. Wainer <xref ref-type="bibr" rid="pone.0027974-Wainer1">[2]</xref>
and Ehrenberg <xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
demonstrate why this is more challenging for the reader to understand. Many tables are burdened with a great volume of numbers, with 36% having more than 100 numbers and some many more (the average is almost exactly 100). The final metric we assessed was the number of digits presented, and here, again, the advice from the literature is for the most part unheeded: 86% of tables tabulate three or more digits, and 39% four or more. There seems little reason for such spurious accuracy, especially since many of these numbers were generated from small Monte Carlo samples, so even if this degree of purported accuracy were helpful to the reader (it isn't), in most situations it would not be warranted on statistical grounds.</p>
<p>There were some differences between the three journals that may interest the dedicated reader: <italic>Statistica Sinica</italic>
generally did worse on typographic issues, tables in the <italic>Journal of the American Statistical Association</italic>
were the most voluminous, while tables in the <italic>Annals of Statistics</italic>
were more likely to be poorly oriented and to have unhelpful overprecision.</p>
</sec>
<sec id="s3b"><title>Cross-over experiment</title>
<p>The results of the experiment showed clear evidence that information could be extracted quicker from graphs than tables (<xref ref-type="fig" rid="pone-0027974-g002">Figure 2</xref>
, left), with the overall average difference in times of 27s (95%I: 14 s, 47 s). Interestingly, the difference between the two media was more pronounced for undergraduate participants–i.e. those with generally less experience of mathematical statistical research–with undergraduates needing around an additional 38 s to answer a question using a table than the equivalent graph (95%I: 17 s, 70 s), i.e. almost double the time, while postgraduates required an additional 19 s (95%I: 7 s, 36 s), around only a 50% increase. Variability within experience groups was low, though present, but variability between the time needed, presumably a reflexion of differences in their inherent difficulty, to extract information from some of the table–graph pairs was substantial.</p>
<fig id="pone-0027974-g002" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0027974.g002</object-id>
<label>Figure 2</label>
<caption><title>Comparison of speed and accuracy at which information can be drawn from tables and figures in randomised cross-over experiment, by experience level.</title>
<p>Distributions in white panels account for parametric uncertainty and variability between individuals and table-graph pairs, and are estimated using standard kernel density estimation from MCMC samples. Distributions in grey panels are for an average individual, an average graph and account only for parameter uncertainty; posterior mean and 95% intervals are plotted. Left: timing in minutes per question. Right: probability of reporting a correct answer. Top: undergraduate statistics students, by medium. Middle: PhD candidates or faculty members, by medium. Bottom: difference between graph and table speed and accuracy of information extraction, by experience.</p>
</caption>
<graphic xlink:href="pone.0027974.g002"></graphic>
</fig>
<p>There was a similar interaction between statistical research experience and the accuracy with which information could be extracted from graphical and tabular displays of simulation studies (<xref ref-type="fig" rid="pone-0027974-g002">Figure 2</xref>
, right). There was no discernible difference between the accuracy of answers derived from tables (80%, 95%I [65,91]%) or graphs (78%, [63,90]%) for the more experienced participants, and the accuracy by which undergraduates could extract information from a graph (82%, [67,93]%) was about the same as that of postgraduates faced with the same graph, but undergraduate participants were markedly less accurate at taking information from tables (60%, [43,77]%; difference 21%, [2,42]%).</p>
<p>The “catch up” effect that allowed less experienced participants to extract information from a graph at a similar speed and accuracy as more experienced participants was also manifested in comparisons between graphical displays read by undergraduates and tabular ones read by postgraduates. There was no discernable difference in accuracy between these experience-stimuli pairs (difference 2%, [<inline-formula><inline-graphic xlink:href="pone.0027974.e023.jpg" mimetype="image"></inline-graphic>
</inline-formula>
,14]%) while there was marginal evidence that the undergraduate armed with a graph could parse it quicker than a table perusing postgraduate (difference 13s, [<inline-formula><inline-graphic xlink:href="pone.0027974.e024.jpg" mimetype="image"></inline-graphic>
</inline-formula>
,33]s).</p>
</sec>
</sec>
<sec id="s4"><title>Discussion</title>
<p>Our results strongly suggest that, to the participants in this experiment, graphical displays provided a more efficient way to communicate the findings of simulation studies than tables. Our readers could parse the graphical equivalent demonstrably faster, and for our undergraduate participants, the graphical medium provided a more accurate understanding of the results of the simulation study (the relatively small sample size prevented us from corroborating if postgraduate participants also had improved accuracy). Had the authors of the source tables switched to a graphical display, less experienced readers would have been able to read and understand the display as quickly and accurately as a reader with postgraduate experience of statistical research, while at the same time allowing the experienced reader to understand the results of the simulation study with less effort.</p>
<p>In reviewing all papers published in three high profile mathematical statistics journals in 2009, we found that many authors continue to present simulation studies in tabular form, and that the prevalence remains high of several characteristics that may impair the ability to understand tables effectively. Such tables are often burdened with too many numbers to be easily understood, are often displayed to too much precision to make sense of or even to make sense (when one considers the often small Monte Carlo sample size from which each element in the table has been derived and the concomitant standard errors), and are often oriented opposite the simplest–vertical–direction in which to compare related numbers. The two design features we quantified that are most easily addressed at the typesetting stage–alignment of decimal places and the preference for unbracketed numbers where possible–were also the two features for which the three journals most consistently adhered to the advice of Ehrenberg <xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
and Wainer <xref ref-type="bibr" rid="pone.0027974-Wainer1">[2]</xref>
, suggesting that authors, not journals, are most culpable for poorly designed tables of simulation studies.</p>
<p>The arguments in favour of graphical presentation of simulation studies are multifold. Graphs provide an easier way to obtain an holistic view of the study, make more forceful statements, and allow specific comparisons more readily, the latter particularly important when multiple methods are contrasted across varying sample sizes and underlying conditions. Wainer <xref ref-type="bibr" rid="pone.0027974-Wainer1">[2]</xref>
suggests four purposes for numerical information, namely <italic>exploration</italic>
, <italic>communication</italic>
, <italic>storage</italic>
and <italic>decoration</italic>
. Our expectation is that of these, only <italic>communication</italic>
and <italic>decoration</italic>
are relevant for simulation studies in research reports: the <italic>exploration</italic>
phase has presumably already been completed by the authors (though graphs may help here, too <xref ref-type="bibr" rid="pone.0027974-Wainer3">[37]</xref>
), who are now seeking to communicate their findings, while <italic>storage</italic>
of cheap Monte Carlo “data” seems irrelevant. Since even badly designed graphs are more eye-catching than ranks of numerical simulation data, we believe that authors who use the table for <italic>decoration</italic>
would be best to switch to a graph. This leaves <italic>communication</italic>
, but the results of our experiment support the argument that this, too, may best be achieved by some kind of graphic.</p>
<p>We anticipate that switching to graphical displays would permit useful structural changes to simulation studies: increasing the number of design points, such as simulated sample sizes, to give a broader picture of the difference between methodologies, and creating more natural opportunities to present uncertainty in Monte Carlo estimates (e.g. via confidence intervals), which currently is, oddly, often neglected in tabular displays. A related curious omission from many tables of simulation studies is a formal statistical analysis of the (simulation) experiment's results; we believe that the results of such analyses of simulated output could gainfully be presented alongside or in place of a graphical rendition of the raw simulation output.</p>
<p>There are of course several limitations to the experimental design that may inhibit the generality of our findings. Study participants were selected from a single environment, namely students and staff of the National University of Singapore, and may consequently differ in subtle ways from consumers of statistical simulation studies in other settings. We purposefully selected individuals with a statistical background, believing them to be the target audience of papers containing statistical simulation studies, and it is unclear to what extent readers with different backgrounds–in particular, with little theoretical statistical training–would differ from the study participants. A further possible limitation is the risk of idiosyncrasies in the particular tables selected and our graphical adaptations thereof: we invested quite a bit of effort in devising suitable graphical presentations of the tables we found, and it is possible (though we hope unlikely) that the authors of the original tables created them without a similar investment. As a reviewer points out, one cannot fairly compare the compositions of Mozart and Lennon when one is performed by a school band and the other a philharmonic orchestra. A fairer assessment of the relative merits of tables and graphs might result from a competition between proponents of the two sides, similar to the recent graphic competition held by <italic>Chance</italic>
magazine <xref ref-type="bibr" rid="pone.0027974-Wainer4">[38]</xref>
. A further limitation is that the performance metrics of time and accuracy related to an artificial task, namely of providing answers to questions that could be quantified as correct or not, though in their natural environment, numerical displays may induce questions that do not yield simplistic, correct/incorrect answers.</p>
<p>The experiment we performed considered design characteristics of tables and graphs at an aggregated level, and thus do not shed light on particular elements of graphical or tabular displays of simulation studies. Although there already exist useful guidelines on how best to design tabular displays <xref ref-type="bibr" rid="pone.0027974-Wainer1">[2]</xref>
, <xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
, <xref ref-type="bibr" rid="pone.0027974-Feinberg1">[6]</xref>
, there is scope for future work to elicit quantitatively the effect of some of the design features in <xref ref-type="fig" rid="pone-0027974-g001">Figure 1</xref>
–such as the volume of numbers in tables or the precision to which they are presented–on the ability of readers to parse information. Such research would allow statistical simulation studies to be designed more efficiently, from the perspective of presenting the results in an intelligible fashion to other researchers, and would also indicate how general the results of this study are. Although we have focused on displays of statistical simulation studies, we expect that the efficiency of displays of data in other fields–such as medicine, in which tables are even more widespread <xref ref-type="bibr" rid="pone.0027974-Feinberg1">[6]</xref>
–would be amenable to future research.</p>
<p>In summary, this study demonstrated that many tables of simulation studies containing undesirable features may be found in the statistical literature, and that–among our study participants at least–these features make the studies harder to understand than a well-constructed graphical equivalent, and may lead to erroneous conclusions for non-experts. Over 30 years ago, Ehrenberg opened his landmark treatise on the rudiments on numeracy thus: <italic>Many tables of data are badly presented</italic>
<xref ref-type="bibr" rid="pone.0027974-Ehrenberg1">[3]</xref>
. It is our opinion that this is still the case and that most, if not all, results of statistical simulation studies should be presented in graphical form.</p>
</sec>
<sec sec-type="supplementary-material" id="s5"><title>Supporting Information</title>
<supplementary-material content-type="local-data" id="pone.0027974.s001"><label>Supporting Information S1</label>
<caption><p><bold>Appendices.</bold>
Appendix 1 contains details of the cross-over experiment, namely the background information provided to participants, the graphical stimuli, and questions asked. Appendix 2 contains details of the prior distributions used in the analysis.</p>
<p>(PDF)</p>
</caption>
<media xlink:href="pone.0027974.s001.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for additional data file.</p>
</caption>
</media>
</supplementary-material>
<supplementary-material content-type="local-data" id="pone.0027974.s002"><label>Supporting Information S2</label>
<caption><p><bold>Data from cross-over experiment.</bold>
</p>
<p>(CSV)</p>
</caption>
<media xlink:href="pone.0027974.s002.csv" mimetype="text" mime-subtype="plain"><caption><p>Click here for additional data file.</p>
</caption>
</media>
</supplementary-material>
<supplementary-material content-type="local-data" id="pone.0027974.s003"><label>Supporting Information S3</label>
<caption><p><bold>Readme file for <xref ref-type="supplementary-material" rid="pone.0027974.s002">Supporting Information S2</xref>
.</bold>
</p>
<p>(TXT)</p>
</caption>
<media xlink:href="pone.0027974.s003.txt" mimetype="text" mime-subtype="plain"><caption><p>Click here for additional data file.</p>
</caption>
</media>
</supplementary-material>
</sec>
</body>
<back><ack><p>We are very grateful to the study participants for kindly providing their time, to Román Carrasco for useful suggestions at preliminary stages of planning the experiments, and to Leontine Alkema and two anonymous reviewers for helpful comments on the manuscript.</p>
</ack>
<fn-group><fn fn-type="conflict"><p><bold>Competing Interests: </bold>
The authors have declared that no competing interests exist.</p>
</fn>
<fn fn-type="financial-disclosure"><p><bold>Funding: </bold>
ARC was supported by a research grant from the National University of Singapore. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</p>
</fn>
</fn-group>
<ref-list><title>References</title>
<ref id="pone.0027974-Farquhar1"><label>1</label>
<element-citation publication-type="book"> <person-group person-group-type="author"><name><surname>Farquhar</surname>
<given-names>AB</given-names>
</name>
<name><surname>Farquhar</surname>
<given-names>H</given-names>
</name>
</person-group>
<year>1891</year>
<article-title>Economic and industrial delusions: a discourse of the case for protection.</article-title>
<publisher-loc>New York</publisher-loc>
<publisher-name>Putnam</publisher-name>
</element-citation>
</ref>
<ref id="pone.0027974-Wainer1"><label>2</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Wainer</surname>
<given-names>H</given-names>
</name>
</person-group>
<year>1997</year>
<article-title>Improving tabular displays, with NAEP tables as examples and inspirations.</article-title>
<source>J Ed Behav Stat</source>
<volume>22</volume>
<fpage>1</fpage>
<lpage>30</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Ehrenberg1"><label>3</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Ehrenberg</surname>
<given-names>ASC</given-names>
</name>
</person-group>
<year>1977</year>
<article-title>Rudiments of numeracy.</article-title>
<source>J Roy Stat Soc A</source>
<volume>140</volume>
<fpage>277</fpage>
<lpage>297</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Mahon1"><label>4</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Mahon</surname>
<given-names>BH</given-names>
</name>
</person-group>
<year>1977</year>
<article-title>Statistics and decisions: the importance of communication and the power of graphical presentation.</article-title>
<source>J Roy Stat Soc A</source>
<volume>140</volume>
<fpage>297</fpage>
<lpage>323</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Gelman1"><label>5</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Gelman</surname>
<given-names>A</given-names>
</name>
<name><surname>Pasarica</surname>
<given-names>C</given-names>
</name>
<name><surname>Dodhia</surname>
<given-names>R</given-names>
</name>
</person-group>
<year>2002</year>
<article-title>Let's practice what we preach: turning tables into graphs.</article-title>
<source>Am Stat</source>
<volume>56</volume>
<fpage>121</fpage>
<lpage>130</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Feinberg1"><label>6</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Feinberg</surname>
<given-names>RA</given-names>
</name>
<name><surname>Wainer</surname>
<given-names>H</given-names>
</name>
</person-group>
<year>2011</year>
<article-title>Extracting sunbeams from cucumbers.</article-title>
<source>J Comp Graph Stat</source>
<volume>20</volume>
</element-citation>
</ref>
<ref id="pone.0027974-Gelman2"><label>7</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Gelman</surname>
<given-names>A</given-names>
</name>
</person-group>
<year>2011</year>
<article-title>Why tables are really much better than graphs.</article-title>
<source>J Comp Graph Stat</source>
<volume>20</volume>
<fpage>3</fpage>
<lpage>7</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Wainer2"><label>8</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Wainer</surname>
<given-names>H</given-names>
</name>
</person-group>
<year>2011</year>
<article-title>Comment: Why tables are really much better than graphs.</article-title>
<source>J Comp Graph Stat</source>
<volume>20</volume>
<fpage>8</fpage>
<lpage>15</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Briggs1"><label>9</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Briggs</surname>
<given-names>WM</given-names>
</name>
</person-group>
<year>2011</year>
<article-title>Comment: Why tables are really much better than graphs.</article-title>
<source>J Comp Graph Stat</source>
<volume>20</volume>
<fpage>16</fpage>
<lpage>17</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Friendly1"><label>10</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Friendly</surname>
<given-names>M</given-names>
</name>
<name><surname>Kwan</surname>
<given-names>E</given-names>
</name>
</person-group>
<year>2011</year>
<article-title>Comment: Why tables are really much better than graphs.</article-title>
<source>J Comp Graph Stat</source>
<volume>20</volume>
<fpage>18</fpage>
<lpage>27</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Wills1"><label>11</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Wills</surname>
<given-names>G</given-names>
</name>
</person-group>
<year>2011</year>
<article-title>Comment: Why tables are really much better than graphs.</article-title>
<source>J Comp Graph Stat</source>
<volume>20</volume>
<fpage>28</fpage>
<lpage>35</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Gelman3"><label>12</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Gelman</surname>
<given-names>A</given-names>
</name>
</person-group>
<year>2011</year>
<article-title>Rejoinder: Why tables are really much better than graphs.</article-title>
<source>J Comp Graph Stat</source>
<volume>20</volume>
<fpage>36</fpage>
<lpage>40</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Remus1"><label>13</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Remus</surname>
<given-names>W</given-names>
</name>
</person-group>
<year>1984</year>
<article-title>An empirical investigation of the impact of graphical and tabular data presenta- tions on decision making.</article-title>
<source>Management Sci</source>
<volume>30</volume>
<fpage>533</fpage>
<lpage>542</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Meyer1"><label>14</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Meyer</surname>
<given-names>J</given-names>
</name>
<name><surname>Shinar</surname>
<given-names>D</given-names>
</name>
<name><surname>Leiser</surname>
<given-names>D</given-names>
</name>
</person-group>
<year>1997</year>
<article-title>Multiple factors that determine performance with tables and graphs.</article-title>
<source>Human Factors</source>
<volume>39</volume>
<fpage>268</fpage>
<lpage>286</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Meyer2"><label>15</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Meyer</surname>
<given-names>J</given-names>
</name>
<name><surname>Shamo</surname>
<given-names>MK</given-names>
</name>
<name><surname>Gopher</surname>
<given-names>D</given-names>
</name>
</person-group>
<year>1999</year>
<article-title>Information structure and the relative e_cacy of tables and graphs.</article-title>
<source>Human Factors</source>
<volume>41</volume>
<fpage>570</fpage>
<lpage>587</lpage>
<pub-id pub-id-type="pmid">10774128</pub-id>
</element-citation>
</ref>
<ref id="pone.0027974-Spence1"><label>16</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Spence</surname>
<given-names>I</given-names>
</name>
<name><surname>Lewandowsky</surname>
<given-names>S</given-names>
</name>
</person-group>
<year>1991</year>
<article-title>Displaying proportions and percentages.</article-title>
<source>Appl Cognitive Psychol</source>
<volume>5</volume>
<fpage>61</fpage>
<lpage>77</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Tufte1"><label>17</label>
<element-citation publication-type="book"> <person-group person-group-type="author"><name><surname>Tufte</surname>
<given-names>ER</given-names>
</name>
</person-group>
<year>2001</year>
<article-title>The Visual Display of Quantitative Information.</article-title>
<publisher-loc>Cheshire, Connecticut</publisher-loc>
<publisher-name>Graphics Press, 2 edition</publisher-name>
</element-citation>
</ref>
<ref id="pone.0027974-Cleveland1"><label>18</label>
<element-citation publication-type="book"> <person-group person-group-type="author"><name><surname>Cleveland</surname>
<given-names>WS</given-names>
</name>
</person-group>
<year>1993</year>
<article-title>Visualizing Data.</article-title>
<publisher-loc>Summit, New Jersey</publisher-loc>
<publisher-name>Hobart Press</publisher-name>
</element-citation>
</ref>
<ref id="pone.0027974-Cleveland2"><label>19</label>
<element-citation publication-type="book"> <person-group person-group-type="author"><name><surname>Cleveland</surname>
<given-names>WS</given-names>
</name>
</person-group>
<year>1994</year>
<article-title>The Elements of Graphing Data.</article-title>
<publisher-loc>Summit, New Jersey</publisher-loc>
<publisher-name>Hobart Press</publisher-name>
</element-citation>
</ref>
<ref id="pone.0027974-Zhang1"><label>20</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Zhang</surname>
<given-names>M</given-names>
</name>
<name><surname>Zhang</surname>
<given-names>D</given-names>
</name>
<name><surname>Wells</surname>
<given-names>MT</given-names>
</name>
</person-group>
<year>2010</year>
<article-title>Generalized thresholding estimators for high-dimensional location parameters.</article-title>
<source>Stat Sinica</source>
<volume>20</volume>
<fpage>911</fpage>
<lpage>926</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Ho1"><label>21</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Ho</surname>
<given-names>YHS</given-names>
</name>
<name><surname>Lee</surname>
<given-names>SMS</given-names>
</name>
</person-group>
<year>2005</year>
<article-title>Iterated smoothed bootstrap confidence intervals for population quantiles.</article-title>
<source>Ann Stat</source>
<volume>33</volume>
<fpage>437</fpage>
<lpage>462</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Conti1"><label>22</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Conti</surname>
<given-names>PL</given-names>
</name>
<name><surname>De Giovanni</surname>
<given-names>L</given-names>
</name>
<name><surname>Stoev</surname>
<given-names>SA</given-names>
</name>
<name><surname>Taqqu</surname>
<given-names>MS</given-names>
</name>
</person-group>
<year>2008</year>
<article-title>Confidence intervals for the long memory parameter based on wavelets and resampling.</article-title>
<source>Stat Sinica</source>
<volume>18</volume>
<fpage>559</fpage>
<lpage>579</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Loh1"><label>23</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Loh</surname>
<given-names>WY</given-names>
</name>
</person-group>
<year>1991</year>
<article-title>Bootstrap calibration for confidence interval construction and selection.</article-title>
<source>Stat Sinica</source>
<volume>1</volume>
<fpage>477</fpage>
<lpage>491</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Roy1"><label>24</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Roy</surname>
<given-names>A</given-names>
</name>
<name><surname>Falk</surname>
<given-names>B</given-names>
</name>
<name><surname>Fuller</surname>
<given-names>WA</given-names>
</name>
</person-group>
<year>2004</year>
<article-title>Testing for trend in the presence of autoregressive error.</article-title>
<source>J Am Stat Assoc</source>
<volume>99</volume>
<fpage>1082</fpage>
<lpage>1091</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Wang1"><label>25</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Wang</surname>
<given-names>C</given-names>
</name>
<name><surname>Paik</surname>
<given-names>MC</given-names>
</name>
</person-group>
<year>2006</year>
<article-title>Efficiencies of methods dealing with missing covariates in regression analysis.</article-title>
<source>Stat Sinica</source>
<volume>16</volume>
<fpage>1169</fpage>
<lpage>1192</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Murrell1"><label>26</label>
<element-citation publication-type="book"> <person-group person-group-type="author"><name><surname>Murrell</surname>
<given-names>P</given-names>
</name>
</person-group>
<year>2005</year>
<article-title>R Graphics.</article-title>
<publisher-loc>London</publisher-loc>
<publisher-name>Chapman Hall/CRC</publisher-name>
</element-citation>
</ref>
<ref id="pone.0027974-R1"><label>27</label>
<element-citation publication-type="journal"> <collab>R Development Core Team</collab>
<year>2011</year>
<article-title>R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.</article-title>
<comment>Available: <ext-link ext-link-type="uri" xlink:href="http://www.R-project.org">http://www.R-project.org</ext-link>
. ISBN 3-900051-07-0</comment>
</element-citation>
</ref>
<ref id="pone.0027974-Gelman4"><label>28</label>
<element-citation publication-type="book"> <person-group person-group-type="author"><name><surname>Gelman</surname>
<given-names>A</given-names>
</name>
<name><surname>Hill</surname>
<given-names>J</given-names>
</name>
</person-group>
<year>2006</year>
<article-title>Data Analysis Using Regression and Multilevel/Hierarchical Models.</article-title>
<publisher-name>Cambridge University Press</publisher-name>
</element-citation>
</ref>
<ref id="pone.0027974-Metropolis1"><label>29</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Metropolis</surname>
<given-names>N</given-names>
</name>
<name><surname>Rosenbluth</surname>
<given-names>AW</given-names>
</name>
<name><surname>Rosenbluth</surname>
<given-names>MN</given-names>
</name>
<name><surname>Teller</surname>
<given-names>AH</given-names>
</name>
<name><surname>Teller</surname>
<given-names>E</given-names>
</name>
</person-group>
<year>1953</year>
<article-title>Equations of state calculations by fast computing machines.</article-title>
<source>J Chem Phys</source>
<volume>21</volume>
<fpage>1087</fpage>
<lpage>1092</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Hastings1"><label>30</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Hastings</surname>
<given-names>WK</given-names>
</name>
</person-group>
<year>1970</year>
<article-title>Monte Carlo sampling methods using Markov chains and their applications.</article-title>
<source>Biometrika</source>
<volume>57</volume>
<fpage>97</fpage>
<lpage>109</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Gelman5"><label>31</label>
<element-citation publication-type="book"> <person-group person-group-type="author"><name><surname>Gelman</surname>
<given-names>A</given-names>
</name>
<name><surname>Carlin</surname>
<given-names>JB</given-names>
</name>
<name><surname>Stern</surname>
<given-names>HS</given-names>
</name>
<name><surname>Rubin</surname>
<given-names>DB</given-names>
</name>
</person-group>
<year>2003</year>
<article-title>Bayesian Data Analysis (2nd ed).</article-title>
<publisher-loc>London</publisher-loc>
<publisher-name>Chapman Hall/CRC</publisher-name>
</element-citation>
</ref>
<ref id="pone.0027974-Gilks1"><label>32</label>
<element-citation publication-type="book"> <person-group person-group-type="author"><name><surname>Gilks</surname>
<given-names>WR</given-names>
</name>
<name><surname>Richardson</surname>
<given-names>S</given-names>
</name>
<name><surname>Spiegelhalter</surname>
<given-names>D</given-names>
</name>
</person-group>
<year>1995</year>
<article-title>Markov Chain Monte Carlo in Practice.</article-title>
<publisher-loc>London</publisher-loc>
<publisher-name>Chapman Hall/CRC</publisher-name>
</element-citation>
</ref>
<ref id="pone.0027974-Thomas1"><label>33</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Thomas</surname>
<given-names>A</given-names>
</name>
<name><surname>O'Hara</surname>
<given-names>B</given-names>
</name>
<name><surname>Ligges</surname>
<given-names>U</given-names>
</name>
<name><surname>Sturtz</surname>
<given-names>S</given-names>
</name>
</person-group>
<year>2006</year>
<article-title>Making bugs open.</article-title>
<source>R News</source>
<volume>6</volume>
<fpage>12</fpage>
<lpage>17</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Gelman6"><label>34</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Gelman</surname>
<given-names>A</given-names>
</name>
<name><surname>Rubin</surname>
<given-names>DB</given-names>
</name>
</person-group>
<year>1992</year>
<article-title>Inference from iterative simulation using multiple sequences.</article-title>
<source>Stat Sci</source>
<volume>7</volume>
<fpage>457</fpage>
<lpage>511</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Brooks1"><label>35</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Brooks</surname>
<given-names>SP</given-names>
</name>
<name><surname>Gelman</surname>
<given-names>A</given-names>
</name>
</person-group>
<year>1997</year>
<article-title>General methods for monitoring convergence of iterative simulations.</article-title>
<source>J Comp Graph Stat</source>
<volume>7</volume>
<fpage>434</fpage>
<lpage>455</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Plummer1"><label>36</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Plummer</surname>
<given-names>M</given-names>
</name>
<name><surname>Best</surname>
<given-names>N</given-names>
</name>
<name><surname>Cowles</surname>
<given-names>K</given-names>
</name>
<name><surname>Vines</surname>
<given-names>K</given-names>
</name>
</person-group>
<year>2010</year>
<article-title>coda: Output analysis and diagnostics for MCMC.</article-title>
<comment>Available: <ext-link ext-link-type="uri" xlink:href="http://CRAN.R-project.org/package=coda">http://CRAN.R-project.org/package=coda</ext-link>
. R package version 0.14-2</comment>
</element-citation>
</ref>
<ref id="pone.0027974-Wainer3"><label>37</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Wainer</surname>
<given-names>H</given-names>
</name>
<name><surname>Lysen</surname>
<given-names>S</given-names>
</name>
</person-group>
<year>2009</year>
<article-title>That's funny. . . .</article-title>
<source>Am Sci</source>
<volume>97</volume>
<fpage>272</fpage>
<lpage>275</lpage>
</element-citation>
</ref>
<ref id="pone.0027974-Wainer4"><label>38</label>
<element-citation publication-type="journal"> <person-group person-group-type="author"><name><surname>Wainer</surname>
<given-names>H</given-names>
</name>
<name><surname>Larsen</surname>
<given-names>M</given-names>
</name>
</person-group>
<year>2009</year>
<article-title>Pictures at an exhibition.</article-title>
<source>Chance</source>
<volume>22</volume>
<fpage>46</fpage>
<lpage>54</lpage>
<pub-id pub-id-type="pmid">20657747</pub-id>
</element-citation>
</ref>
</ref-list>
</back>
</pmc>
<affiliations><list></list>
<tree><noCountry><name sortKey="Cook, Alex R" sort="Cook, Alex R" uniqKey="Cook A" first="Alex R." last="Cook">Alex R. Cook</name>
<name sortKey="Teo, Shanice W L" sort="Teo, Shanice W L" uniqKey="Teo S" first="Shanice W. L." last="Teo">Shanice W. L. Teo</name>
</noCountry>
</tree>
</affiliations>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Musique/explor/MozartV1/Data/Pmc/Checkpoint
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000313 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Checkpoint/biblio.hfd -nk 000313 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Wicri/Musique |area= MozartV1 |flux= Pmc |étape= Checkpoint |type= RBID |clé= PMC:3223202 |texte= The Communicability of Graphical Alternatives to Tabular Displays of Statistical Simulation Studies }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Checkpoint/RBID.i -Sk "pubmed:22132184" \ | HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Checkpoint/biblio.hfd \ | NlmPubMed2Wicri -a MozartV1
![]() | This area was generated with Dilib version V0.6.20. | ![]() |