Systematic review automation technologies
Identifieur interne : 000037 ( Pmc/Checkpoint ); précédent : 000036; suivant : 000038Systematic review automation technologies
Auteurs : Guy Tsafnat [Australie] ; Paul Glasziou [Australie] ; Miew Keen Choong [Australie] ; Adam Dunn [Australie] ; Filippo Galgani [Australie] ; Enrico Coiera [Australie]Source :
- Systematic Reviews [ 2046-4053 ] ; 2014.
Abstract
Systematic reviews, a cornerstone of evidence-based medicine, are not produced quickly enough to support clinical practice. The cost of production, availability of the requisite expertise and timeliness are often quoted as major contributors for the delay. This detailed survey of the state of the art of information systems designed to support or automate individual tasks in the systematic review, and in particular systematic reviews of randomized controlled clinical trials, reveals trends that see the convergence of several parallel research projects.
We surveyed literature describing informatics systems that support or automate the processes of systematic review or each of the tasks of the systematic review. Several projects focus on automating, simplifying and/or streamlining specific tasks of the systematic review. Some tasks are already fully automated while others are still largely manual. In this review, we describe each task and the effect that its automation would have on the entire systematic review process, summarize the existing information system support for each task, and highlight where further research is needed for realizing automation for the task. Integration of the systems that automate systematic review tasks may lead to a revised systematic review workflow. We envisage the optimized workflow will lead to system in which each systematic review is described as a computer program that automatically retrieves relevant trials, appraises them, extracts and synthesizes data, evaluates the risk of bias, performs meta-analysis calculations, and produces a report in real time.
Url:
DOI: 10.1186/2046-4053-3-74
PubMed: 25005128
PubMed Central: 4100748
Affiliations:
Links toward previous steps (curation, corpus...)
Links to Exploration step
PMC:4100748Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Systematic review automation technologies</title><author><name sortKey="Tsafnat, Guy" sort="Tsafnat, Guy" uniqKey="Tsafnat G" first="Guy" last="Tsafnat">Guy Tsafnat</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author><author><name sortKey="Glasziou, Paul" sort="Glasziou, Paul" uniqKey="Glasziou P" first="Paul" last="Glasziou">Paul Glasziou</name><affiliation wicri:level="1"><nlm:aff id="I2">Centre for Research on Evidence Based Practice, Bond University, Gold Coast, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Research on Evidence Based Practice, Bond University, Gold Coast</wicri:regionArea><wicri:noRegion>Gold Coast</wicri:noRegion></affiliation></author><author><name sortKey="Choong, Miew Keen" sort="Choong, Miew Keen" uniqKey="Choong M" first="Miew Keen" last="Choong">Miew Keen Choong</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author><author><name sortKey="Dunn, Adam" sort="Dunn, Adam" uniqKey="Dunn A" first="Adam" last="Dunn">Adam Dunn</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author><author><name sortKey="Galgani, Filippo" sort="Galgani, Filippo" uniqKey="Galgani F" first="Filippo" last="Galgani">Filippo Galgani</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author><author><name sortKey="Coiera, Enrico" sort="Coiera, Enrico" uniqKey="Coiera E" first="Enrico" last="Coiera">Enrico Coiera</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">PMC</idno><idno type="pmid">25005128</idno><idno type="pmc">4100748</idno><idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4100748</idno><idno type="RBID">PMC:4100748</idno><idno type="doi">10.1186/2046-4053-3-74</idno><date when="2014">2014</date><idno type="wicri:Area/Pmc/Corpus">000110</idno><idno type="wicri:Area/Pmc/Curation">000110</idno><idno type="wicri:Area/Pmc/Checkpoint">000037</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Systematic review automation technologies</title><author><name sortKey="Tsafnat, Guy" sort="Tsafnat, Guy" uniqKey="Tsafnat G" first="Guy" last="Tsafnat">Guy Tsafnat</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author><author><name sortKey="Glasziou, Paul" sort="Glasziou, Paul" uniqKey="Glasziou P" first="Paul" last="Glasziou">Paul Glasziou</name><affiliation wicri:level="1"><nlm:aff id="I2">Centre for Research on Evidence Based Practice, Bond University, Gold Coast, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Research on Evidence Based Practice, Bond University, Gold Coast</wicri:regionArea><wicri:noRegion>Gold Coast</wicri:noRegion></affiliation></author><author><name sortKey="Choong, Miew Keen" sort="Choong, Miew Keen" uniqKey="Choong M" first="Miew Keen" last="Choong">Miew Keen Choong</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author><author><name sortKey="Dunn, Adam" sort="Dunn, Adam" uniqKey="Dunn A" first="Adam" last="Dunn">Adam Dunn</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author><author><name sortKey="Galgani, Filippo" sort="Galgani, Filippo" uniqKey="Galgani F" first="Filippo" last="Galgani">Filippo Galgani</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author><author><name sortKey="Coiera, Enrico" sort="Coiera, Enrico" uniqKey="Coiera E" first="Enrico" last="Coiera">Enrico Coiera</name><affiliation wicri:level="3"><nlm:aff id="I1">Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</nlm:aff><country xml:lang="fr">Australie</country><wicri:regionArea>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney</wicri:regionArea><placeName><settlement type="city">Sydney</settlement></placeName></affiliation></author></analytic><series><title level="j">Systematic Reviews</title><idno type="eISSN">2046-4053</idno><imprint><date when="2014">2014</date></imprint></series></biblStruct></sourceDesc></fileDesc><profileDesc><textClass></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en"><p>Systematic reviews, a cornerstone of evidence-based medicine, are not produced quickly enough to support clinical practice. The cost of production, availability of the requisite expertise and timeliness are often quoted as major contributors for the delay. This detailed survey of the state of the art of information systems designed to support or automate individual tasks in the systematic review, and in particular systematic reviews of randomized controlled clinical trials, reveals trends that see the convergence of several parallel research projects.</p><p>We surveyed literature describing informatics systems that support or automate the processes of systematic review or each of the tasks of the systematic review. Several projects focus on automating, simplifying and/or streamlining specific tasks of the systematic review. Some tasks are already fully automated while others are still largely manual. In this review, we describe each task and the effect that its automation would have on the entire systematic review process, summarize the existing information system support for each task, and highlight where further research is needed for realizing automation for the task. Integration of the systems that automate systematic review tasks may lead to a revised systematic review workflow. We envisage the optimized workflow will lead to system in which each systematic review is described as a computer program that automatically retrieves relevant trials, appraises them, extracts and synthesizes data, evaluates the risk of bias, performs meta-analysis calculations, and produces a report in real time.</p></div></front><back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Sackett, Dl" uniqKey="Sackett D">DL Sackett</name></author><author><name sortKey="Straus, S" uniqKey="Straus S">S Straus</name></author><author><name sortKey="Richardson, Ws" uniqKey="Richardson W">WS Richardson</name></author><author><name sortKey="Rosenberg, W" uniqKey="Rosenberg W">W Rosenberg</name></author><author><name sortKey="Haynes, Rb" uniqKey="Haynes R">RB Haynes</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cochrane, Al" uniqKey="Cochrane A">AL Cochrane</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Jaidee, W" uniqKey="Jaidee W">W Jaidee</name></author><author><name sortKey="Moher, D" uniqKey="Moher D">D Moher</name></author><author><name sortKey="Laopaiboon, M" uniqKey="Laopaiboon M">M Laopaiboon</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bastian, H" uniqKey="Bastian H">H Bastian</name></author><author><name sortKey="Glasziou, P" uniqKey="Glasziou P">P Glasziou</name></author><author><name sortKey="Chalmers, I" uniqKey="Chalmers I">I Chalmers</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Tsafnat, G" uniqKey="Tsafnat G">G Tsafnat</name></author><author><name sortKey="Dunn, A" uniqKey="Dunn A">A Dunn</name></author><author><name sortKey="Glasziou, P" uniqKey="Glasziou P">P Glasziou</name></author><author><name sortKey="Coiera, E" uniqKey="Coiera E">E Coiera</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Smith, R" uniqKey="Smith R">R Smith</name></author><author><name sortKey="Chalmers, I" uniqKey="Chalmers I">I Chalmers</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Shojania, Kg" uniqKey="Shojania K">KG Shojania</name></author><author><name sortKey="Sampson, M" uniqKey="Sampson M">M Sampson</name></author><author><name sortKey="Ansari, Mt" uniqKey="Ansari M">MT Ansari</name></author><author><name sortKey="Ji, J" uniqKey="Ji J">J Ji</name></author><author><name sortKey="Doucette, S" uniqKey="Doucette S">S Doucette</name></author><author><name sortKey="Moher, D" uniqKey="Moher D">D Moher</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Takwoingi, Y" uniqKey="Takwoingi Y">Y Takwoingi</name></author><author><name sortKey="Hopewell, S" uniqKey="Hopewell S">S Hopewell</name></author><author><name sortKey="Tovey, D" uniqKey="Tovey D">D Tovey</name></author><author><name sortKey="Sutton, Aj" uniqKey="Sutton A">AJ Sutton</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Adams, Ce" uniqKey="Adams C">CE Adams</name></author><author><name sortKey="Polzmacher, S" uniqKey="Polzmacher S">S Polzmacher</name></author><author><name sortKey="Wolff, A" uniqKey="Wolff A">A Wolff</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Glasziou, P" uniqKey="Glasziou P">P Glasziou</name></author><author><name sortKey="Beller, E" uniqKey="Beller E">E Beller</name></author><author><name sortKey="Thorning, S" uniqKey="Thorning S">S Thorning</name></author><author><name sortKey="Vermeulen, M" uniqKey="Vermeulen M">M Vermeulen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Rennels, Gd" uniqKey="Rennels G">GD Rennels</name></author><author><name sortKey="Shortliffe, Eh" uniqKey="Shortliffe E">EH Shortliffe</name></author><author><name sortKey="Stockdale, Fe" uniqKey="Stockdale F">FE Stockdale</name></author><author><name sortKey="Miller, Pl" uniqKey="Miller P">PL Miller</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wallace, Bc" uniqKey="Wallace B">BC Wallace</name></author><author><name sortKey="Dahabreh, Ij" uniqKey="Dahabreh I">IJ Dahabreh</name></author><author><name sortKey="Schmid, Ch" uniqKey="Schmid C">CH Schmid</name></author><author><name sortKey="Lau, J" uniqKey="Lau J">J Lau</name></author><author><name sortKey="Trikalinos, Ta" uniqKey="Trikalinos T">TA Trikalinos</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Coiera, E" uniqKey="Coiera E">E Coiera</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cohen, Am" uniqKey="Cohen A">AM Cohen</name></author><author><name sortKey="Adams, Ce" uniqKey="Adams C">CE Adams</name></author><author><name sortKey="Davis, Jm" uniqKey="Davis J">JM Davis</name></author><author><name sortKey="Yu, C" uniqKey="Yu C">C Yu</name></author><author><name sortKey="Yu, Ps" uniqKey="Yu P">PS Yu</name></author><author><name sortKey="Meng, W" uniqKey="Meng W">W Meng</name></author><author><name sortKey="Duggan, L" uniqKey="Duggan L">L Duggan</name></author><author><name sortKey="Mcdonagh, M" uniqKey="Mcdonagh M">M McDonagh</name></author><author><name sortKey="Smalheiser, Nr" uniqKey="Smalheiser N">NR Smalheiser</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Counsell, C" uniqKey="Counsell C">C Counsell</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Oxman, Ad" uniqKey="Oxman A">AD Oxman</name></author><author><name sortKey="Sackett, Dl" uniqKey="Sackett D">DL Sackett</name></author><author><name sortKey="Guyatt, Gh" uniqKey="Guyatt G">GH Guyatt</name></author><author><name sortKey="Group, E Bmw" uniqKey="Group E">E-BMW Group</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bragge, P" uniqKey="Bragge P">P Bragge</name></author><author><name sortKey="Clavisi, O" uniqKey="Clavisi O">O Clavisi</name></author><author><name sortKey="Turner, T" uniqKey="Turner T">T Turner</name></author><author><name sortKey="Tavender, E" uniqKey="Tavender E">E Tavender</name></author><author><name sortKey="Collie, A" uniqKey="Collie A">A Collie</name></author><author><name sortKey="Gruen, Rl" uniqKey="Gruen R">RL Gruen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Snilstveit, B" uniqKey="Snilstveit B">B Snilstveit</name></author><author><name sortKey="Vojtkova, M" uniqKey="Vojtkova M">M Vojtkova</name></author><author><name sortKey="Bhavsar, A" uniqKey="Bhavsar A">A Bhavsar</name></author><author><name sortKey="Gaarder, M" uniqKey="Gaarder M">M Gaarder</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Arksey, H" uniqKey="Arksey H">H Arksey</name></author><author><name sortKey="O Alley, L" uniqKey="O Alley L">L O’Malley</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bero, L" uniqKey="Bero L">L Bero</name></author><author><name sortKey="Busuttil, G" uniqKey="Busuttil G">G Busuttil</name></author><author><name sortKey="Farquhar, C" uniqKey="Farquhar C">C Farquhar</name></author><author><name sortKey="Koehlmoos, Tp" uniqKey="Koehlmoos T">TP Koehlmoos</name></author><author><name sortKey="Moher, D" uniqKey="Moher D">D Moher</name></author><author><name sortKey="Nylenna, M" uniqKey="Nylenna M">M Nylenna</name></author><author><name sortKey="Smith, R" uniqKey="Smith R">R Smith</name></author><author><name sortKey="Tovey, D" uniqKey="Tovey D">D Tovey</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Smalheiser, Nr" uniqKey="Smalheiser N">NR Smalheiser</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Malhotra, A" uniqKey="Malhotra A">A Malhotra</name></author><author><name sortKey="Younesi, E" uniqKey="Younesi E">E Younesi</name></author><author><name sortKey="Gurulingappa, H" uniqKey="Gurulingappa H">H Gurulingappa</name></author><author><name sortKey="Hofmann Apitius, M" uniqKey="Hofmann Apitius M">M Hofmann-Apitius</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cohen, T" uniqKey="Cohen T">T Cohen</name></author><author><name sortKey="Whitfield, Gk" uniqKey="Whitfield G">GK Whitfield</name></author><author><name sortKey="Schvaneveldt, Rw" uniqKey="Schvaneveldt R">RW Schvaneveldt</name></author><author><name sortKey="Mukund, K" uniqKey="Mukund K">K Mukund</name></author><author><name sortKey="Rindflesch, T" uniqKey="Rindflesch T">T Rindflesch</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Murray, Cj" uniqKey="Murray C">CJ Murray</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Chalmers, I" uniqKey="Chalmers I">I Chalmers</name></author><author><name sortKey="Glasziou, P" uniqKey="Glasziou P">P Glasziou</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Booth, A" uniqKey="Booth A">A Booth</name></author><author><name sortKey="Clarke, M" uniqKey="Clarke M">M Clarke</name></author><author><name sortKey="Ghersi, D" uniqKey="Ghersi D">D Ghersi</name></author><author><name sortKey="Moher, D" uniqKey="Moher D">D Moher</name></author><author><name sortKey="Petticrew, M" uniqKey="Petticrew M">M Petticrew</name></author><author><name sortKey="Stewart, L" uniqKey="Stewart L">L Stewart</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Carini, S" uniqKey="Carini S">S Carini</name></author><author><name sortKey="Sim, I" uniqKey="Sim I">I Sim</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Glanville, J" uniqKey="Glanville J">J Glanville</name></author><author><name sortKey="Bayliss, S" uniqKey="Bayliss S">S Bayliss</name></author><author><name sortKey="Booth, A" uniqKey="Booth A">A Booth</name></author><author><name sortKey="Dundar, Y" uniqKey="Dundar Y">Y Dundar</name></author><author><name sortKey="Fernandes, H" uniqKey="Fernandes H">H Fernandes</name></author><author><name sortKey="Fleeman, Nd" uniqKey="Fleeman N">ND Fleeman</name></author><author><name sortKey="Foster, L" uniqKey="Foster L">L Foster</name></author><author><name sortKey="Fraser, C" uniqKey="Fraser C">C Fraser</name></author><author><name sortKey="Fry Smith, A" uniqKey="Fry Smith A">A Fry-Smith</name></author><author><name sortKey="Golder, S" uniqKey="Golder S">S Golder</name></author><author><name sortKey="Lefebvre, C" uniqKey="Lefebvre C">C Lefebvre</name></author><author><name sortKey="Miller, C" uniqKey="Miller C">C Miller</name></author><author><name sortKey="Paisley, S" uniqKey="Paisley S">S Paisley</name></author><author><name sortKey="Payne, L" uniqKey="Payne L">L Payne</name></author><author><name sortKey="Price, A" uniqKey="Price A">A Price</name></author><author><name sortKey="Welch, K" uniqKey="Welch K">K Welch</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wong, Ss" uniqKey="Wong S">SS Wong</name></author><author><name sortKey="Wilczynski, Nl" uniqKey="Wilczynski N">NL Wilczynski</name></author><author><name sortKey="Haynes, Rb" uniqKey="Haynes R">RB Haynes</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wilczynski, Nl" uniqKey="Wilczynski N">NL Wilczynski</name></author><author><name sortKey="Haynes, Rb" uniqKey="Haynes R">RB Haynes</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Montori, Vm" uniqKey="Montori V">VM Montori</name></author><author><name sortKey="Wilczynski, Nl" uniqKey="Wilczynski N">NL Wilczynski</name></author><author><name sortKey="Morgan, D" uniqKey="Morgan D">D Morgan</name></author><author><name sortKey="Haynes, Rb" uniqKey="Haynes R">RB Haynes</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wilczynski, Nl" uniqKey="Wilczynski N">NL Wilczynski</name></author><author><name sortKey="Haynes, Rb" uniqKey="Haynes R">RB Haynes</name></author><author><name sortKey="Lavis, Jn" uniqKey="Lavis J">JN Lavis</name></author><author><name sortKey="Ramkissoonsingh, R" uniqKey="Ramkissoonsingh R">R Ramkissoonsingh</name></author><author><name sortKey="Arnold Oatley, Ae" uniqKey="Arnold Oatley A">AE Arnold-Oatley</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Haynes, Rb" uniqKey="Haynes R">RB Haynes</name></author><author><name sortKey="Wilczynski, Nl" uniqKey="Wilczynski N">NL Wilczynski</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Shojania, Kg" uniqKey="Shojania K">KG Shojania</name></author><author><name sortKey="Bero, La" uniqKey="Bero L">LA Bero</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wilczynski, Nl" uniqKey="Wilczynski N">NL Wilczynski</name></author><author><name sortKey="Mckibbon, Ka" uniqKey="Mckibbon K">KA McKibbon</name></author><author><name sortKey="Haynes, Rb" uniqKey="Haynes R">RB Haynes</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Simmons, Rf" uniqKey="Simmons R">RF Simmons</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kwok, C" uniqKey="Kwok C">C Kwok</name></author><author><name sortKey="Etzioni, O" uniqKey="Etzioni O">O Etzioni</name></author><author><name sortKey="Weld, Ds" uniqKey="Weld D">DS Weld</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Dumais, S" uniqKey="Dumais S">S Dumais</name></author><author><name sortKey="Banko, M" uniqKey="Banko M">M Banko</name></author><author><name sortKey="Brill, E" uniqKey="Brill E">E Brill</name></author><author><name sortKey="Lin, J" uniqKey="Lin J">J Lin</name></author><author><name sortKey="Ng, A" uniqKey="Ng A">A Ng</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Collaboration, Rtc" uniqKey="Collaboration R">RTC Collaboration</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Higgins, J" uniqKey="Higgins J">J Higgins</name></author><author><name sortKey="Churchill, R" uniqKey="Churchill R">R Churchill</name></author><author><name sortKey="Tovey, D" uniqKey="Tovey D">D Tovey</name></author><author><name sortKey="Lasserson, T" uniqKey="Lasserson T">T Lasserson</name></author><author><name sortKey="Chandler, J" uniqKey="Chandler J">J Chandler</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Tsafnat, G" uniqKey="Tsafnat G">G Tsafnat</name></author><author><name sortKey="Coiera, E" uniqKey="Coiera E">E Coiera</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Sim, I" uniqKey="Sim I">I Sim</name></author><author><name sortKey="Detmer, De" uniqKey="Detmer D">DE Detmer</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Sim, I" uniqKey="Sim I">I Sim</name></author><author><name sortKey="Tu, Sw" uniqKey="Tu S">SW Tu</name></author><author><name sortKey="Carini, S" uniqKey="Carini S">S Carini</name></author><author><name sortKey="Lehmann, Hp" uniqKey="Lehmann H">HP Lehmann</name></author><author><name sortKey="Pollock, Bh" uniqKey="Pollock B">BH Pollock</name></author><author><name sortKey="Peleg, M" uniqKey="Peleg M">M Peleg</name></author><author><name sortKey="Wittkowski, Km" uniqKey="Wittkowski K">KM Wittkowski</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Balk, Em" uniqKey="Balk E">EM Balk</name></author><author><name sortKey="Chung, M" uniqKey="Chung M">M Chung</name></author><author><name sortKey="Chen, Ml" uniqKey="Chen M">ML Chen</name></author><author><name sortKey="Chang, Lkw" uniqKey="Chang L">LKW Chang</name></author><author><name sortKey="Trikalinos, Ta" uniqKey="Trikalinos T">TA Trikalinos</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Greenhalgh, T" uniqKey="Greenhalgh T">T Greenhalgh</name></author><author><name sortKey="Peacock, R" uniqKey="Peacock R">R Peacock</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kuper, H" uniqKey="Kuper H">H Kuper</name></author><author><name sortKey="Nicholson, A" uniqKey="Nicholson A">A Nicholson</name></author><author><name sortKey="Hemingway, H" uniqKey="Hemingway H">H Hemingway</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cepeda, Ms" uniqKey="Cepeda M">MS Cepeda</name></author><author><name sortKey="Lobanov, V" uniqKey="Lobanov V">V Lobanov</name></author><author><name sortKey="Berlin, Ja" uniqKey="Berlin J">JA Berlin</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Lowe, Hj" uniqKey="Lowe H">HJ Lowe</name></author><author><name sortKey="Barnett, Go" uniqKey="Barnett G">GO Barnett</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Robinson, Ka" uniqKey="Robinson K">KA Robinson</name></author><author><name sortKey="Dickersin, K" uniqKey="Dickersin K">K Dickersin</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Doshi, P" uniqKey="Doshi P">P Doshi</name></author><author><name sortKey="Jones, M" uniqKey="Jones M">M Jones</name></author><author><name sortKey="Jefferson, T" uniqKey="Jefferson T">T Jefferson</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Eysenbach, G" uniqKey="Eysenbach G">G Eysenbach</name></author><author><name sortKey="Tuische, J" uniqKey="Tuische J">J Tuische</name></author><author><name sortKey="Diepgen, Tl" uniqKey="Diepgen T">TL Diepgen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Mckibbon, Ka" uniqKey="Mckibbon K">KA McKibbon</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Harris, Mr" uniqKey="Harris M">MR Harris</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Beller, Em" uniqKey="Beller E">EM Beller</name></author><author><name sortKey="Chen, Jk" uniqKey="Chen J">JK Chen</name></author><author><name sortKey="Wang, Ul" uniqKey="Wang U">UL Wang</name></author><author><name sortKey="Glasziou, Pp" uniqKey="Glasziou P">PP Glasziou</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hull, D" uniqKey="Hull D">D Hull</name></author><author><name sortKey="Pettifer, Sr" uniqKey="Pettifer S">SR Pettifer</name></author><author><name sortKey="Kell, Db" uniqKey="Kell D">DB Kell</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Chalmers, I" uniqKey="Chalmers I">I Chalmers</name></author><author><name sortKey="Glasziou, P" uniqKey="Glasziou P">P Glasziou</name></author><author><name sortKey="Godlee, F" uniqKey="Godlee F">F Godlee</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Huser, V" uniqKey="Huser V">V Huser</name></author><author><name sortKey="Cimino, Jj" uniqKey="Cimino J">JJ Cimino</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Wilkins, T" uniqKey="Wilkins T">T Wilkins</name></author><author><name sortKey="Gillies, Ra" uniqKey="Gillies R">RA Gillies</name></author><author><name sortKey="Davies, K" uniqKey="Davies K">K Davies</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wolf, F" uniqKey="Wolf F">F Wolf</name></author><author><name sortKey="Grum, C" uniqKey="Grum C">C Grum</name></author><author><name sortKey="Bara, A" uniqKey="Bara A">A Bara</name></author><author><name sortKey="Milan, S" uniqKey="Milan S">S Milan</name></author><author><name sortKey="Jones, P" uniqKey="Jones P">P Jones</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Woods, D" uniqKey="Woods D">D Woods</name></author><author><name sortKey="Trewheellar, K" uniqKey="Trewheellar K">K Trewheellar</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Shariff, Sz" uniqKey="Shariff S">SZ Shariff</name></author><author><name sortKey="Bejaimal, Sa" uniqKey="Bejaimal S">SA Bejaimal</name></author><author><name sortKey="Sontrop, Jm" uniqKey="Sontrop J">JM Sontrop</name></author><author><name sortKey="Iansavichus, Av" uniqKey="Iansavichus A">AV Iansavichus</name></author><author><name sortKey="Haynes, Rb" uniqKey="Haynes R">RB Haynes</name></author><author><name sortKey="Weir, Ma" uniqKey="Weir M">MA Weir</name></author><author><name sortKey="Garg, Ax" uniqKey="Garg A">AX Garg</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Carpineto, C" uniqKey="Carpineto C">C Carpineto</name></author><author><name sortKey="Romano, G" uniqKey="Romano G">G Romano</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Chatterjee, R" uniqKey="Chatterjee R">R Chatterjee</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Boyack, Kw" uniqKey="Boyack K">KW Boyack</name></author><author><name sortKey="Newman, D" uniqKey="Newman D">D Newman</name></author><author><name sortKey="Duhon, Rj" uniqKey="Duhon R">RJ Duhon</name></author><author><name sortKey="Klavans, R" uniqKey="Klavans R">R Klavans</name></author><author><name sortKey="Patek, M" uniqKey="Patek M">M Patek</name></author><author><name sortKey="Biberstine, Jr" uniqKey="Biberstine J">JR Biberstine</name></author><author><name sortKey="Schijvenaars, B" uniqKey="Schijvenaars B">B Schijvenaars</name></author><author><name sortKey="Skupin, A" uniqKey="Skupin A">A Skupin</name></author><author><name sortKey="Ma, N" uniqKey="Ma N">N Ma</name></author><author><name sortKey="Borner, K" uniqKey="Borner K">K Börner</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Aljaber, B" uniqKey="Aljaber B">B Aljaber</name></author><author><name sortKey="Stokes, N" uniqKey="Stokes N">N Stokes</name></author><author><name sortKey="Bailey, J" uniqKey="Bailey J">J Bailey</name></author><author><name sortKey="Pei, J" uniqKey="Pei J">J Pei</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Smalheiser, Nr" uniqKey="Smalheiser N">NR Smalheiser</name></author><author><name sortKey="Lin, C" uniqKey="Lin C">C Lin</name></author><author><name sortKey="Jia, L" uniqKey="Jia L">L Jia</name></author><author><name sortKey="Jiang, Y" uniqKey="Jiang Y">Y Jiang</name></author><author><name sortKey="Cohen, Am" uniqKey="Cohen A">AM Cohen</name></author><author><name sortKey="Yu, C" uniqKey="Yu C">C Yu</name></author><author><name sortKey="Davis, Jm" uniqKey="Davis J">JM Davis</name></author><author><name sortKey="Adams, Ce" uniqKey="Adams C">CE Adams</name></author><author><name sortKey="Mcdonagh, Ms" uniqKey="Mcdonagh M">MS McDonagh</name></author><author><name sortKey="Meng, W" uniqKey="Meng W">W Meng</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bracke, Pj" uniqKey="Bracke P">PJ Bracke</name></author><author><name sortKey="Howse, Dk" uniqKey="Howse D">DK Howse</name></author><author><name sortKey="Keim, Sm" uniqKey="Keim S">SM Keim</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Coiera, E" uniqKey="Coiera E">E Coiera</name></author><author><name sortKey="Walther, M" uniqKey="Walther M">M Walther</name></author><author><name sortKey="Nguyen, K" uniqKey="Nguyen K">K Nguyen</name></author><author><name sortKey="Lovell, Nh" uniqKey="Lovell N">NH Lovell</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Magrabi, F" uniqKey="Magrabi F">F Magrabi</name></author><author><name sortKey="Westbrook, Ji" uniqKey="Westbrook J">JI Westbrook</name></author><author><name sortKey="Kidd, Mr" uniqKey="Kidd M">MR Kidd</name></author><author><name sortKey="Day, Ro" uniqKey="Day R">RO Day</name></author><author><name sortKey="Coiera, E" uniqKey="Coiera E">E Coiera</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Meats, E" uniqKey="Meats E">E Meats</name></author><author><name sortKey="Brassey, J" uniqKey="Brassey J">J Brassey</name></author><author><name sortKey="Heneghan, C" uniqKey="Heneghan C">C Heneghan</name></author><author><name sortKey="Glasziou, P" uniqKey="Glasziou P">P Glasziou</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Whetzel, Pl" uniqKey="Whetzel P">PL Whetzel</name></author><author><name sortKey="Noy, Nf" uniqKey="Noy N">NF Noy</name></author><author><name sortKey="Shah, Nh" uniqKey="Shah N">NH Shah</name></author><author><name sortKey="Alexander, Pr" uniqKey="Alexander P">PR Alexander</name></author><author><name sortKey="Nyulas, C" uniqKey="Nyulas C">C Nyulas</name></author><author><name sortKey="Tudorache, T" uniqKey="Tudorache T">T Tudorache</name></author><author><name sortKey="Musen, Ma" uniqKey="Musen M">MA Musen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="De Angelis, C" uniqKey="De Angelis C">C De Angelis</name></author><author><name sortKey="Drazen, Jm" uniqKey="Drazen J">JM Drazen</name></author><author><name sortKey="Frizelle, Fa" uniqKey="Frizelle F">FA Frizelle</name></author><author><name sortKey="Haug, C" uniqKey="Haug C">C Haug</name></author><author><name sortKey="Hoey, J" uniqKey="Hoey J">J Hoey</name></author><author><name sortKey="Horton, R" uniqKey="Horton R">R Horton</name></author><author><name sortKey="Kotzin, S" uniqKey="Kotzin S">S Kotzin</name></author><author><name sortKey="Laine, C" uniqKey="Laine C">C Laine</name></author><author><name sortKey="Marusic, A" uniqKey="Marusic A">A Marusic</name></author><author><name sortKey="Overbeke, Ajp" uniqKey="Overbeke A">AJP Overbeke</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Askie, Lm" uniqKey="Askie L">LM Askie</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Qi, X S" uniqKey="Qi X">X-S Qi</name></author><author><name sortKey="Bai, M" uniqKey="Bai M">M Bai</name></author><author><name sortKey="Yang, Z P" uniqKey="Yang Z">Z-P Yang</name></author><author><name sortKey="Ren, W R" uniqKey="Ren W">W-R Ren</name></author><author><name sortKey="Mcfarland, Lv" uniqKey="Mcfarland L">LV McFarland</name></author><author><name sortKey="Goh, S" uniqKey="Goh S">S Goh</name></author><author><name sortKey="Linz, K" uniqKey="Linz K">K Linz</name></author><author><name sortKey="Miller, Bj" uniqKey="Miller B">BJ Miller</name></author><author><name sortKey="Zhang, H" uniqKey="Zhang H">H Zhang</name></author><author><name sortKey="Gao, C" uniqKey="Gao C">C Gao</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Qi, X" uniqKey="Qi X">X Qi</name></author><author><name sortKey="Yang, M" uniqKey="Yang M">M Yang</name></author><author><name sortKey="Ren, W" uniqKey="Ren W">W Ren</name></author><author><name sortKey="Jia, J" uniqKey="Jia J">J Jia</name></author><author><name sortKey="Wang, J" uniqKey="Wang J">J Wang</name></author><author><name sortKey="Han, G" uniqKey="Han G">G Han</name></author><author><name sortKey="Fan, D" uniqKey="Fan D">D Fan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Elmagarmid, Ak" uniqKey="Elmagarmid A">AK Elmagarmid</name></author><author><name sortKey="Ipeirotis, Pg" uniqKey="Ipeirotis P">PG Ipeirotis</name></author><author><name sortKey="Verykios, Vs" uniqKey="Verykios V">VS Verykios</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Aabenhus, R" uniqKey="Aabenhus R">R Aabenhus</name></author><author><name sortKey="Jensen, Ju" uniqKey="Jensen J">JU Jensen</name></author><author><name sortKey="Cals, Jw" uniqKey="Cals J">JW Cals</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Researchsoft, I" uniqKey="Researchsoft I">I ResearchSoft</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Jiang, Y" uniqKey="Jiang Y">Y Jiang</name></author><author><name sortKey="Lin, C" uniqKey="Lin C">C Lin</name></author><author><name sortKey="Meng, W" uniqKey="Meng W">W Meng</name></author><author><name sortKey="Yu, C" uniqKey="Yu C">C Yu</name></author><author><name sortKey="Cohen, Am" uniqKey="Cohen A">AM Cohen</name></author><author><name sortKey="Smalheiser, Nr" uniqKey="Smalheiser N">NR Smalheiser</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Councill, Ig" uniqKey="Councill I">IG Councill</name></author><author><name sortKey="Giles, Cl" uniqKey="Giles C">CL Giles</name></author><author><name sortKey="Kan, M Y" uniqKey="Kan M">M-Y Kan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Granitzer, M" uniqKey="Granitzer M">M Granitzer</name></author><author><name sortKey="Hristakeva, M" uniqKey="Hristakeva M">M Hristakeva</name></author><author><name sortKey="Jack, K" uniqKey="Jack K">K Jack</name></author><author><name sortKey="Knight, R" uniqKey="Knight R">R Knight</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Nhat, Hdh" uniqKey="Nhat H">HDH Nhat</name></author><author><name sortKey="Bysani, P" uniqKey="Bysani P">P Bysani</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Saleem, O" uniqKey="Saleem O">O Saleem</name></author><author><name sortKey="Latif, S" uniqKey="Latif S">S Latif</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Day, M Y" uniqKey="Day M">M-Y Day</name></author><author><name sortKey="Tsai, T H" uniqKey="Tsai T">T-H Tsai</name></author><author><name sortKey="Sung, C L" uniqKey="Sung C">C-L Sung</name></author><author><name sortKey="Lee, C W" uniqKey="Lee C">C-W Lee</name></author><author><name sortKey="Wu, S H" uniqKey="Wu S">S-H Wu</name></author><author><name sortKey="Ong, C S" uniqKey="Ong C">C-S Ong</name></author><author><name sortKey="Hsu, W L" uniqKey="Hsu W">W-L Hsu</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Errami, M" uniqKey="Errami M">M Errami</name></author><author><name sortKey="Hicks, Jm" uniqKey="Hicks J">JM Hicks</name></author><author><name sortKey="Fisher, W" uniqKey="Fisher W">W Fisher</name></author><author><name sortKey="Trusty, D" uniqKey="Trusty D">D Trusty</name></author><author><name sortKey="Wren, Jd" uniqKey="Wren J">JD Wren</name></author><author><name sortKey="Long, Tc" uniqKey="Long T">TC Long</name></author><author><name sortKey="Garner, Hr" uniqKey="Garner H">HR Garner</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Zhang, Hy" uniqKey="Zhang H">HY Zhang</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Meuschke, N" uniqKey="Meuschke N">N Meuschke</name></author><author><name sortKey="Gipp, B" uniqKey="Gipp B">B Gipp</name></author><author><name sortKey="Breitinger, C" uniqKey="Breitinger C">C Breitinger</name></author><author><name sortKey="Berkeley, U" uniqKey="Berkeley U">U Berkeley</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Chung, Gy" uniqKey="Chung G">GY Chung</name></author><author><name sortKey="Coiera, E" uniqKey="Coiera E">E Coiera</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Thomas, J" uniqKey="Thomas J">J Thomas</name></author><author><name sortKey="Mcnaught, J" uniqKey="Mcnaught J">J McNaught</name></author><author><name sortKey="Ananiadou, S" uniqKey="Ananiadou S">S Ananiadou</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kiritchenko, S" uniqKey="Kiritchenko S">S Kiritchenko</name></author><author><name sortKey="De Bruijn, B" uniqKey="De Bruijn B">B de Bruijn</name></author><author><name sortKey="Carini, S" uniqKey="Carini S">S Carini</name></author><author><name sortKey="Martin, J" uniqKey="Martin J">J Martin</name></author><author><name sortKey="Sim, I" uniqKey="Sim I">I Sim</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kim, Sn" uniqKey="Kim S">SN Kim</name></author><author><name sortKey="Martinez, D" uniqKey="Martinez D">D Martinez</name></author><author><name sortKey="Cavedon, L" uniqKey="Cavedon L">L Cavedon</name></author><author><name sortKey="Yencken, L" uniqKey="Yencken L">L Yencken</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Zhao, J" uniqKey="Zhao J">J Zhao</name></author><author><name sortKey="Bysani, P" uniqKey="Bysani P">P Bysani</name></author><author><name sortKey="Kan, My" uniqKey="Kan M">MY Kan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hansen, Mj" uniqKey="Hansen M">MJ Hansen</name></author><author><name sortKey="Rasmussen, N" uniqKey="Rasmussen N">NØ Rasmussen</name></author><author><name sortKey="Chung, G" uniqKey="Chung G">G Chung</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Thomas, J" uniqKey="Thomas J">J Thomas</name></author><author><name sortKey="Brunton, J" uniqKey="Brunton J">J Brunton</name></author><author><name sortKey="Graziosi, S" uniqKey="Graziosi S">S Graziosi</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Garcia Adeva, Jj" uniqKey="Garcia Adeva J">JJ García Adeva</name></author><author><name sortKey="Pikatza Atxa, Jm" uniqKey="Pikatza Atxa J">JM Pikatza Atxa</name></author><author><name sortKey="Ubeda Carrillo, M" uniqKey="Ubeda Carrillo M">M Ubeda Carrillo</name></author><author><name sortKey="Ansuategi Zengotitabengoa, E" uniqKey="Ansuategi Zengotitabengoa E">E Ansuategi Zengotitabengoa</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wallace, Bc" uniqKey="Wallace B">BC Wallace</name></author><author><name sortKey="Trikalinos, Ta" uniqKey="Trikalinos T">TA Trikalinos</name></author><author><name sortKey="Lau, J" uniqKey="Lau J">J Lau</name></author><author><name sortKey="Brodley, C" uniqKey="Brodley C">C Brodley</name></author><author><name sortKey="Schmid, Ch" uniqKey="Schmid C">CH Schmid</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wallace, Bc" uniqKey="Wallace B">BC Wallace</name></author><author><name sortKey="Small, K" uniqKey="Small K">K Small</name></author><author><name sortKey="Brodley, Ce" uniqKey="Brodley C">CE Brodley</name></author><author><name sortKey="Lau, J" uniqKey="Lau J">J Lau</name></author><author><name sortKey="Trikalinos, Ta" uniqKey="Trikalinos T">TA Trikalinos</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Ananiadou, S" uniqKey="Ananiadou S">S Ananiadou</name></author><author><name sortKey="Rea, B" uniqKey="Rea B">B Rea</name></author><author><name sortKey="Okazaki, N" uniqKey="Okazaki N">N Okazaki</name></author><author><name sortKey="Procter, R" uniqKey="Procter R">R Procter</name></author><author><name sortKey="Thomas, J" uniqKey="Thomas J">J Thomas</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cohen, Am" uniqKey="Cohen A">AM Cohen</name></author><author><name sortKey="Ambert, K" uniqKey="Ambert K">K Ambert</name></author><author><name sortKey="Mcdonagh, M" uniqKey="Mcdonagh M">M McDonagh</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Sarker, A" uniqKey="Sarker A">A Sarker</name></author><author><name sortKey="Molla Aliod, D" uniqKey="Molla Aliod D">D Mollá-Aliod</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kim, Mh" uniqKey="Kim M">MH Kim</name></author><author><name sortKey="Compton, P" uniqKey="Compton P">P Compton</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Rodriguez Esteban, R" uniqKey="Rodriguez Esteban R">R Rodriguez-Esteban</name></author><author><name sortKey="Iossifov, I" uniqKey="Iossifov I">I Iossifov</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Ceri, S" uniqKey="Ceri S">S Ceri</name></author><author><name sortKey="Bozzon, A" uniqKey="Bozzon A">A Bozzon</name></author><author><name sortKey="Brambilla, M" uniqKey="Brambilla M">M Brambilla</name></author><author><name sortKey="Della Valle, E" uniqKey="Della Valle E">E Della Valle</name></author><author><name sortKey="Fraternali, P" uniqKey="Fraternali P">P Fraternali</name></author><author><name sortKey="Quarteroni, S" uniqKey="Quarteroni S">S Quarteroni</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Horsley, T" uniqKey="Horsley T">T Horsley</name></author><author><name sortKey="Dingwall, O" uniqKey="Dingwall O">O Dingwall</name></author><author><name sortKey="Sampson, M" uniqKey="Sampson M">M Sampson</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Robinson, Ka" uniqKey="Robinson K">KA Robinson</name></author><author><name sortKey="Dunn, A" uniqKey="Dunn A">A Dunn</name></author><author><name sortKey="Tsafnat, G" uniqKey="Tsafnat G">G Tsafnat</name></author><author><name sortKey="Glasziou, P" uniqKey="Glasziou P">P Glasziou</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Summerscales, R" uniqKey="Summerscales R">R Summerscales</name></author><author><name sortKey="Argamon, S" uniqKey="Argamon S">S Argamon</name></author><author><name sortKey="Hupert, J" uniqKey="Hupert J">J Hupert</name></author><author><name sortKey="Schwartz, A" uniqKey="Schwartz A">A Schwartz</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Summerscales, Rl" uniqKey="Summerscales R">RL Summerscales</name></author><author><name sortKey="Argamon, S" uniqKey="Argamon S">S Argamon</name></author><author><name sortKey="Bai, S" uniqKey="Bai S">S Bai</name></author><author><name sortKey="Huperff, J" uniqKey="Huperff J">J Huperff</name></author><author><name sortKey="Schwartzff, A" uniqKey="Schwartzff A">A Schwartzff</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hsu, W" uniqKey="Hsu W">W Hsu</name></author><author><name sortKey="Speier, W" uniqKey="Speier W">W Speier</name></author><author><name sortKey="Taira, Rk" uniqKey="Taira R">RK Taira</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Huwaldt, Ja" uniqKey="Huwaldt J">JA Huwaldt</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Egner, J" uniqKey="Egner J">J Egner</name></author><author><name sortKey="Mitchell, M" uniqKey="Mitchell M">M Mitchell</name></author><author><name sortKey="Richter, T" uniqKey="Richter T">T Richter</name></author><author><name sortKey="Winchen, T" uniqKey="Winchen T">T Winchen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Rohatgi, A" uniqKey="Rohatgi A">A Rohatgi</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Imai, E" uniqKey="Imai E">E Imai</name></author><author><name sortKey="Chan, Jc" uniqKey="Chan J">JC Chan</name></author><author><name sortKey="Ito, S" uniqKey="Ito S">S Ito</name></author><author><name sortKey="Yamasaki, T" uniqKey="Yamasaki T">T Yamasaki</name></author><author><name sortKey="Kobayashi, F" uniqKey="Kobayashi F">F Kobayashi</name></author><author><name sortKey="Haneda, M" uniqKey="Haneda M">M Haneda</name></author><author><name sortKey="Makino, H" uniqKey="Makino H">H Makino</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Qahtani, Nh" uniqKey="Qahtani N">NH Qahtani</name></author><author><name sortKey="Hajeri, Fa" uniqKey="Hajeri F">FA Hajeri</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Chung, Gy" uniqKey="Chung G">GY Chung</name></author><author><name sortKey="Coiera, E" uniqKey="Coiera E">E Coiera</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Richard, J" uniqKey="Richard J">J Richard</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hedges, Lv" uniqKey="Hedges L">LV Hedges</name></author><author><name sortKey="Olkin, I" uniqKey="Olkin I">I Olkin</name></author><author><name sortKey="Statistiker, M" uniqKey="Statistiker M">M Statistiker</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Lewis, S" uniqKey="Lewis S">S Lewis</name></author><author><name sortKey="Clarke, M" uniqKey="Clarke M">M Clarke</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Egger, M" uniqKey="Egger M">M Egger</name></author><author><name sortKey="Smith, Gd" uniqKey="Smith G">GD Smith</name></author><author><name sortKey="Schneider, M" uniqKey="Schneider M">M Schneider</name></author><author><name sortKey="Minder, C" uniqKey="Minder C">C Minder</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wallace, Bc" uniqKey="Wallace B">BC Wallace</name></author><author><name sortKey="Schmid, Ch" uniqKey="Schmid C">CH Schmid</name></author><author><name sortKey="Lau, J" uniqKey="Lau J">J Lau</name></author><author><name sortKey="Trikalinos, Ta" uniqKey="Trikalinos T">TA Trikalinos</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Zamora, J" uniqKey="Zamora J">J Zamora</name></author><author><name sortKey="Abraira, V" uniqKey="Abraira V">V Abraira</name></author><author><name sortKey="Muriel, A" uniqKey="Muriel A">A Muriel</name></author><author><name sortKey="Khan, K" uniqKey="Khan K">K Khan</name></author><author><name sortKey="Coomarasamy, A" uniqKey="Coomarasamy A">A Coomarasamy</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Rosenberg, Ms" uniqKey="Rosenberg M">MS Rosenberg</name></author><author><name sortKey="Adams, Dc" uniqKey="Adams D">DC Adams</name></author><author><name sortKey="Gurevitch, J" uniqKey="Gurevitch J">J Gurevitch</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bax, L" uniqKey="Bax L">L Bax</name></author><author><name sortKey="Yu, L M" uniqKey="Yu L">L-M Yu</name></author><author><name sortKey="Ikeda, N" uniqKey="Ikeda N">N Ikeda</name></author><author><name sortKey="Moons, Kg" uniqKey="Moons K">KG Moons</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Moher, D" uniqKey="Moher D">D Moher</name></author><author><name sortKey="Liberati, A" uniqKey="Liberati A">A Liberati</name></author><author><name sortKey="Tetzlaff, J" uniqKey="Tetzlaff J">J Tetzlaff</name></author><author><name sortKey="Altman, Dg" uniqKey="Altman D">DG Altman</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Reiter, E" uniqKey="Reiter E">E Reiter</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Elliott, Jh" uniqKey="Elliott J">JH Elliott</name></author><author><name sortKey="Turner, T" uniqKey="Turner T">T Turner</name></author><author><name sortKey="Clavisi, O" uniqKey="Clavisi O">O Clavisi</name></author><author><name sortKey="Thomas, J" uniqKey="Thomas J">J Thomas</name></author><author><name sortKey="Higgins, Jp" uniqKey="Higgins J">JP Higgins</name></author><author><name sortKey="Mavergames, C" uniqKey="Mavergames C">C Mavergames</name></author><author><name sortKey="Gruen, Rl" uniqKey="Gruen R">RL Gruen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kawamoto, K" uniqKey="Kawamoto K">K Kawamoto</name></author><author><name sortKey="Honey, A" uniqKey="Honey A">A Honey</name></author><author><name sortKey="Rubin, K" uniqKey="Rubin K">K Rubin</name></author></analytic></biblStruct></listBibl></div1></back></TEI><pmc article-type="research-article" xml:lang="en"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">Syst Rev</journal-id><journal-id journal-id-type="iso-abbrev">Syst Rev</journal-id><journal-title-group><journal-title>Systematic Reviews</journal-title></journal-title-group><issn pub-type="epub">2046-4053</issn><publisher><publisher-name>BioMed Central</publisher-name></publisher></journal-meta><article-meta><article-id pub-id-type="pmid">25005128</article-id><article-id pub-id-type="pmc">4100748</article-id><article-id pub-id-type="publisher-id">2046-4053-3-74</article-id><article-id pub-id-type="doi">10.1186/2046-4053-3-74</article-id><article-categories><subj-group subj-group-type="heading"><subject>Commentary</subject></subj-group></article-categories><title-group><article-title>Systematic review automation technologies</article-title></title-group><contrib-group><contrib contrib-type="author" corresp="yes" id="A1"><name><surname>Tsafnat</surname><given-names>Guy</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>guyt@unsw.edu.au</email></contrib><contrib contrib-type="author" id="A2"><name><surname>Glasziou</surname><given-names>Paul</given-names></name><xref ref-type="aff" rid="I2">2</xref><email>pglaszio@bond.edu.au</email></contrib><contrib contrib-type="author" id="A3"><name><surname>Choong</surname><given-names>Miew Keen</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>m.choong@unsw.edu.au</email></contrib><contrib contrib-type="author" id="A4"><name><surname>Dunn</surname><given-names>Adam</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>a.dunn@unsw.edu.au</email></contrib><contrib contrib-type="author" id="A5"><name><surname>Galgani</surname><given-names>Filippo</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>filippo@unsw.edu.au</email></contrib><contrib contrib-type="author" id="A6"><name><surname>Coiera</surname><given-names>Enrico</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>e.coiera@unsw.edu.au</email></contrib></contrib-group><aff id="I1"><label>1</label>Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Sydney, Australia</aff><aff id="I2"><label>2</label>Centre for Research on Evidence Based Practice, Bond University, Gold Coast, Australia</aff><pub-date pub-type="collection"><year>2014</year></pub-date><pub-date pub-type="epub"><day>9</day><month>7</month><year>2014</year></pub-date><volume>3</volume><fpage>74</fpage><lpage>74</lpage><history><date date-type="received"><day>12</day><month>3</month><year>2014</year></date><date date-type="accepted"><day>26</day><month>6</month><year>2014</year></date></history><permissions><copyright-statement>Copyright © 2014 Tsafnat et al.; licensee BioMed Central Ltd.</copyright-statement><copyright-year>2014</copyright-year><copyright-holder>Tsafnat et al.; licensee BioMed Central Ltd.</copyright-holder><license license-type="open-access" xlink:href="http://creativecommons.org/licenses/by/4.0"><license-p>This is an Open Access article distributed under the terms of the Creative Commons Attribution License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0">http://creativecommons.org/licenses/by/4.0</ext-link>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/publicdomain/zero/1.0/">http://creativecommons.org/publicdomain/zero/1.0/</ext-link>) applies to the data made available in this article, unless otherwise stated.</license-p></license></permissions><self-uri xlink:href="http://www.systematicreviewsjournal.com/content/3/1/74"></self-uri><abstract><p>Systematic reviews, a cornerstone of evidence-based medicine, are not produced quickly enough to support clinical practice. The cost of production, availability of the requisite expertise and timeliness are often quoted as major contributors for the delay. This detailed survey of the state of the art of information systems designed to support or automate individual tasks in the systematic review, and in particular systematic reviews of randomized controlled clinical trials, reveals trends that see the convergence of several parallel research projects.</p><p>We surveyed literature describing informatics systems that support or automate the processes of systematic review or each of the tasks of the systematic review. Several projects focus on automating, simplifying and/or streamlining specific tasks of the systematic review. Some tasks are already fully automated while others are still largely manual. In this review, we describe each task and the effect that its automation would have on the entire systematic review process, summarize the existing information system support for each task, and highlight where further research is needed for realizing automation for the task. Integration of the systems that automate systematic review tasks may lead to a revised systematic review workflow. We envisage the optimized workflow will lead to system in which each systematic review is described as a computer program that automatically retrieves relevant trials, appraises them, extracts and synthesizes data, evaluates the risk of bias, performs meta-analysis calculations, and produces a report in real time.</p></abstract><kwd-group><kwd>Systematic reviews</kwd><kwd>Process automation</kwd><kwd>Information retrieval</kwd><kwd>Information extraction</kwd></kwd-group></article-meta></front><body><sec><title>Background</title><p>Evidence-based medicine stipulates that all relevant evidence be used to make clinical decisions regardless of the implied resource demands [<xref ref-type="bibr" rid="B1">1</xref>]. Systematic reviews were invented as a means to enable clinicians to use evidence-based medicine [<xref ref-type="bibr" rid="B2">2</xref>]. However, even the conduct and upkeep of updated systematic reviews required to answer a significant proportion of clinical questions, is beyond our means without automation [<xref ref-type="bibr" rid="B3">3</xref>-<xref ref-type="bibr" rid="B9">9</xref>].</p><p>Systematic reviews are conducted through a robust but slow and resource-intensive process. The result is that undertaking a systematic review may require a large amount of resources and can take years [<xref ref-type="bibr" rid="B10">10</xref>]. Proposed decision support systems for systematic reviewers include ones that help the basic tasks of systematic reviews [<xref ref-type="bibr" rid="B6">6</xref>,<xref ref-type="bibr" rid="B12">12</xref>,<xref ref-type="bibr" rid="B13">13</xref>]. The full automation of systematic reviews that will deliver the best evidence at the right time to the point of care is the logical next step [<xref ref-type="bibr" rid="B14">14</xref>]. Indeed, research into automatic systematic review systems is distinct from - and related to - research on systematic reviews [<xref ref-type="bibr" rid="B5">5</xref>].</p><p>The process of creating a systematic review can be broken down into between four and 15 tasks depending on resolution [<xref ref-type="bibr" rid="B5">5</xref>,<xref ref-type="bibr" rid="B10">10</xref>,<xref ref-type="bibr" rid="B13">13</xref>,<xref ref-type="bibr" rid="B15">15</xref>]. Figure <xref ref-type="fig" rid="F1">1</xref> shows both high-level phases and high-resolution tasks encompassed by the phases. The process is not as linear and compartmentalized as is suggested by only the high-level view. In this article, we review each task, and its role in automatic systematic reviews. We have focused on systematic reviews of randomized controlled trials but most of the tasks will apply to other systematic reviews. Some tasks are not amenable to automation, or have not been examined for the potential to be automated; decision support tools for these tasks are considered instead. In the last section of this paper, we describe a systematic review development environment in which all manual tasks are performed as part of a definition stage of the review, and then a software agent automatically creates and checks the review according to these definitions when needed and at a push of a button.</p><fig id="F1" position="float"><label>Figure 1</label><caption><p><bold>Existing methods for systematic reviews follow these steps with some variations.</bold> Not all systematic reviews follow all steps. This process typically takes between 12 and 24 months. Adapted from the Cochrane [<xref ref-type="bibr" rid="B10">10</xref>] and CREBP [<xref ref-type="bibr" rid="B11">11</xref>] Manuals for systematic reviews. <italic>SR</italic> systematic review, <italic>SD</italic> standard deviation.</p></caption><graphic xlink:href="2046-4053-3-74-1"></graphic></fig></sec><sec sec-type="discussion"><title>Discussion</title><sec><title>What should and should not be automated</title><p>The process of designing a systematic review is in part creative and part technical. We note that systematic review protocols have a natural dichotomy for tasks: creative tasks are done during the development of the question and protocol and technical tasks can be performed automatically and exactly according to the protocol. Thus, the development of the review question(s) is where creativity, experience, and judgment are to be used. Often, the protocol is peer-reviewed to ensure its objectivity and fulfillment of the review question [<xref ref-type="bibr" rid="B10">10</xref>]. Conducting the review should then be a matter of following the review protocol as accurately and objectively as possible.</p><p>In this scenario, the review protocol is developed much like a recipe that can then be executed by machine [<xref ref-type="bibr" rid="B5">5</xref>]. Tasks are reordered so that necessarily manual tasks are shifted to the start of the review, and automatic tasks follow. As an example, consider risk of bias assessment which sometimes requires judgment depending on the outcome measures, the intervention and the research question. During the review protocol preparation, a reviewer would train the system to make the required specific judgment heuristics for that systematic review. A classification engine will use these judgments later in the review to appraise papers. Updating a review becomes a matter of executing the review at the time that it is needed. This frees systematic reviewers to shift their focus from the tedious tasks which are automatable, to the creative tasks of developing the review protocol where human intuition, expertise, and common sense are needed and providing intelligent interpretations of the collected studies. In addition, the reviewers will monitor and assure the quality of the execution to ensure the overall standard of the review.</p><p>The automation of some tasks may seem impossible and fantastic. However, tools dedicated to the automation of evidence synthesis tasks serve as evidence that what seemed fantastic only a few decades ago is now a reality. The development of such tools is incremental and inevitably, limitations are, and will remain, part of the process. A few examples are given in Table <xref ref-type="table" rid="T1">1</xref>.</p><table-wrap position="float" id="T1"><label>Table 1</label><caption><p>Examples of tools used for the automation of evidence synthesis tasks</p></caption><table frame="hsides" rules="groups" border="1"><colgroup><col align="left"></col><col align="left"></col><col align="left"></col><col align="left"></col></colgroup><thead valign="top"><tr><th align="left"><bold>Step</bold></th><th align="left"><bold>Example application</bold></th><th align="left"><bold>Description</bold></th><th align="left"><bold>Limitations</bold></th></tr></thead><tbody valign="top"><tr><td align="left" valign="bottom">Search<hr></hr></td><td align="left" valign="bottom">Quick Clinical<hr></hr></td><td align="left" valign="bottom">Federated meta-search engine<hr></hr></td><td align="left" valign="bottom">Limited source databases not optimized for systematic reviews<hr></hr></td></tr><tr><td align="left" valign="bottom">Search<hr></hr></td><td align="left" valign="bottom">Sherlock<hr></hr></td><td align="left" valign="bottom">Search engine for trial registries<hr></hr></td><td align="left" valign="bottom">Limited to clinicaltrials.gov<hr></hr></td></tr><tr><td align="left" valign="bottom">Search<hr></hr></td><td align="left" valign="bottom">Metta<hr></hr></td><td align="left" valign="bottom">Federated meta-search engine for SR<hr></hr></td><td align="left" valign="bottom">Not available publicly<hr></hr></td></tr><tr><td align="left" valign="bottom">Snowballing<hr></hr></td><td align="left" valign="bottom">ParsCit<hr></hr></td><td align="left" valign="bottom">Reference string extraction from published papers<hr></hr></td><td align="left" valign="bottom">Does not fetch nor recursively pursue citations<hr></hr></td></tr><tr><td align="left" valign="bottom">Screen titles and abstracts<hr></hr></td><td align="left" valign="bottom">Abstrackr<hr></hr></td><td align="left" valign="bottom">Machine learning -based abstract screening tool<hr></hr></td><td align="left" valign="bottom">May reduce review recall by up to 5%<hr></hr></td></tr><tr><td align="left" valign="bottom">Extract data<hr></hr></td><td align="left" valign="bottom">ExaCT<hr></hr></td><td align="left" valign="bottom">PICO and other information element extraction from abstracts<hr></hr></td><td align="left" valign="bottom">No association (e.g. of outcome with trial arm), results only available in HTML<hr></hr></td></tr><tr><td align="left" valign="bottom">Extract data<hr></hr></td><td align="left" valign="bottom">WebPlotDigitizer<hr></hr></td><td align="left" valign="bottom">Re-digitization of data from graphs and plots.<hr></hr></td><td align="left" valign="bottom">No support for survival curves, no optical character recognition<hr></hr></td></tr><tr><td align="left" valign="bottom">Meta-analyze<hr></hr></td><td align="left" valign="bottom">Meta-analyst<hr></hr></td><td align="left" valign="bottom">Create a meta-analysis from extracted data<hr></hr></td><td align="left" valign="bottom">Limited integration with data-extraction and conversion programs<hr></hr></td></tr><tr><td align="left" valign="bottom">Write-up<hr></hr></td><td align="left" valign="bottom">RevMan-HAL<hr></hr></td><td align="left" valign="bottom">Automatic summary write-up from extracted data.<hr></hr></td><td align="left" valign="bottom">Only works with RevMan files<hr></hr></td></tr><tr><td align="left">Write-up</td><td align="left">PRISMA Flow Diagram Generator</td><td align="left">Automatic generation of PRISMA diagrams</td><td align="left">Does not support some complex diagrams</td></tr></tbody></table></table-wrap></sec><sec><title>Finding decision support tools for systematic reviewers</title><p>We have searched PubMed, Scopus, and Google Scholar for papers that describe informatics systems that support the process of systematic reviews, meta-analysis, other related evidence surveying process, automation of each of the tasks of the systematic review. We have used a broad definition of automation that includes software that streamline processes by automating even only the trivial parts of a task such as automatically populating database fields with manually extracted data. We excluded manually constructed databases even if an application programming interface is provided, although we did not necessarily exclude automatic tools that make use of such databases. We searched for each systematic review task names, synonymous terms as well as names of the algorithms used in papers on automation and systematic reviews. We tracked citations both forward and backward and included a representative set of papers that describe the state of the art.</p></sec><sec><title>How individual tasks are supported</title><p>This section describes each of the systematic review tasks in Figure <xref ref-type="fig" rid="F1">1</xref> in detail. Each task is described along with the potential benefits from its automation. State-of-the-art systems that automate or support the task are listed and the next stages of research are highlighted.</p><sec><title>Task 1: formulate the review question</title><sec><title>Task description</title><p>There is no single correct way to formulate review questions [<xref ref-type="bibr" rid="B16">16</xref>]; although, it is desirable to prioritize review questions by burden of disease for which there is a lack of review in the area. While choosing a review topic is not constrained, factors such as expertise in the area and personal interest are common [<xref ref-type="bibr" rid="B16">16</xref>]. All research questions need to be presented with sufficient detail so as to reduce ambiguity and help with the critical appraisal of trials that will be included or excluded from the review. The population, intervention, control, and outcome (PICO) were recommended as elements that should be included in any question [<xref ref-type="bibr" rid="B1">1</xref>,<xref ref-type="bibr" rid="B17">17</xref>]. A well-written review question provides logical and unambiguous criteria for inclusion or exclusion of trials.</p></sec><sec><title>Automation potential</title><p>Automatic systems can help identify missing evidence, support creative processes, and support the creative processes of identifying questions of personal interest and expertise. Prioritization of questions can save effort that would otherwise be spent on unimportant, irrelevant or uninteresting questions and duplication. Decision support for question formulation can ensure questions are fully and unambiguously specified before the review is started.</p></sec><sec><title>Current systems</title><p>Global evidence maps [<xref ref-type="bibr" rid="B18">18</xref>,<xref ref-type="bibr" rid="B19">19</xref>] and scoping studies [<xref ref-type="bibr" rid="B20">20</xref>] are both literature review methods designed to complement systematic review and identify research gaps. Both differ from systematic review in that they seek to address broader questions about particular areas rather than answer specific questions answered by a narrow set of clinical trials. Scoping studies are relatively quick reviews of the area. Global evidence maps are conducted in a formal process, similar to systematic reviews, and thus may take in excess of 2 years [<xref ref-type="bibr" rid="B21">21</xref>]. Broad reviews direct reviewers towards questions for which there is a gap in the evidence. Algorithms that can extract the evidence gaps identified in such reviews can provide decision support to systematic reviewers choosing a research question.</p></sec><sec><title>Future research</title><p>Research for additional decision support tools should focus on identification of new problems. Work in the field of artificial intelligence on hypothesis generation and finding [<xref ref-type="bibr" rid="B22">22</xref>-<xref ref-type="bibr" rid="B24">24</xref>] may provide an automatic means of suggesting novel review questions. Economic modeling tools and databases that can automatically assess burden of disease according to established standards [<xref ref-type="bibr" rid="B25">25</xref>] can help prioritize potential review questions.</p><p>Question prioritization might not be required when systematic reviews can be conducted quickly and economically. An exhaustive set of research questions can be asked about the suitability of any treatment for any population. While the number of combinations of questions and answers poses its own challenges, automatic synthesis will quickly determine that most condition-treatment pairs have no evidence (e.g. because they are nonsensical) and will not dedicate much time to them.</p></sec></sec><sec><title>Task 2: find previous systematic reviews</title><sec><title>Task description</title><p>The resources and time required to produce a systematic review are so great that independently creating a duplicate of an existing review is both wasteful and avoidable [<xref ref-type="bibr" rid="B26">26</xref>]. Therefore, if the reviewer can find a systematic review that answers the same question, the potential savings might be substantial. However, finding a previous systematic review is not trivial and could be improved.</p></sec><sec><title>Automation potential</title><p>An accurate and automatic system that finds previous systematic reviews given a clinical question will directly reduce the effort needed to establish whether a previous review exists. Indirectly, such a system can also reduce the number of redundant reviews. Even if an out-of-date systematic review is identified, updating it according to an established protocol is preferable to conducting a new review.</p></sec><sec><title>Current systems</title><p>Databases of systematic reviews include the Cochrane Database of Systematic Reviews [<xref ref-type="bibr" rid="B6">6</xref>], Database of Abstracts of Reviews of Effects (DARE), the International Register of Prospective Systematic Review (PROSPERO) [<xref ref-type="bibr" rid="B27">27</xref>], SysBank [<xref ref-type="bibr" rid="B28">28</xref>], and PubMed Health. These should be searched before a new review is started.</p><p>Search strategies and filters have been suggested to improve recall (the proportion of relevant documents retrieved out of all relevant documents) and precision (the proportion of relevant documents retrieved out of all retrieved documents) respectively of search queries [<xref ref-type="bibr" rid="B29">29</xref>-<xref ref-type="bibr" rid="B34">34</xref>]. Specific search filters designed to find systematic reviews have been proposed [<xref ref-type="bibr" rid="B29">29</xref>,<xref ref-type="bibr" rid="B35">35</xref>]. PubMed's Clinical Queries feature adds a filter [<xref ref-type="bibr" rid="B32">32</xref>] that restricts searches to clinical trials and systematic reviews. PubMed Health provides a search filter that works well for finding systematic reviews but is limited to PubMed and DARE currently [<xref ref-type="bibr" rid="B36">36</xref>].</p><p>Automation of this task is to find systematic reviews that answer the research question without manually translating research questions into search queries. Question answering systems have been in development since the 1960s [<xref ref-type="bibr" rid="B37">37</xref>]. For this task, a form of an automatic web-based question-answering system [<xref ref-type="bibr" rid="B38">38</xref>,<xref ref-type="bibr" rid="B39">39</xref>] in which the answer to the review question is a systematic review, is most suitable.</p></sec><sec><title>Future research</title><p>Research is needed on the design and evaluation of specialized question-answering systems that accept the nuances of systematic review questions, use search filters and strategies to search multiple databases to find the relevant systematic review or systematic review protocol.</p></sec></sec><sec><title>Task 3: write the protocol</title><sec><title>Task description</title><p>Writing the systematic review protocol is the first step in planning the review once it is established that the review is needed and does not already exist. This task requires specialized expertise in medicine, library science, clinical standards and statistics. This task also requires creativity and close familiarity with the literature in the topic because reviewers must imagine the outcomes in order to design the questions and aims. As there are currently no methods to formally review the consistency of the protocol, peer-review is used to ensure that the proposed protocol is complete, has the potential to answer the review question, and is unbiased.</p></sec><sec><title>Automation potential</title><p>Writing the review protocol formally will allow it to be automatically checked for consistency and logical integrity. This would ensure that the protocol is consistent, unbiased, and appropriate for the research question. Verified protocols can thus reduce or remove the need for protocol peer review altogether.</p></sec><sec><title>Current systems</title><p>Systems that are used to support the writing of systematic review protocols include templates and macros. Cochrane's Review Manager [<xref ref-type="bibr" rid="B40">40</xref>] uses a protocol template: it has standard fields that remind the reviewer to cover all aspects of a Cochrane protocol, including inclusion criteria, search methods, appraisal, extraction, and statistical analysis plan. Improvement of the templates is subject to ongoing research [<xref ref-type="bibr" rid="B40">40</xref>,<xref ref-type="bibr" rid="B41">41</xref>].</p></sec><sec><title>Future research</title><p>Automation and support for protocol writing can include reasoning logic that checks the completeness of the inclusion criteria (e.g., by checking that the population, intervention, and outcome are all specified), warns of potential biases in the inclusion criteria (e.g., that the disease being researched is more prevalent in different age groups but the protocol does not account for age), and tests the consistency and feasibility of the inclusion criteria (e.g., warns that a population of women with prostate cancer is probably wrong).</p><p>This sort of consistency verification is a type of computational reasoning task that requires models of the problem to be created through knowledge representation, simulation, and/or constraint resolution [<xref ref-type="bibr" rid="B42">42</xref>]. Such models for clinical trials were proposed in the 1980s [<xref ref-type="bibr" rid="B43">43</xref>], are still evolving [<xref ref-type="bibr" rid="B44">44</xref>], and similar models should also be possible for other clinical questions.</p><p>Language bias remains an unanswered problem with little existing systems available to address it [<xref ref-type="bibr" rid="B45">45</xref>]. Automatic information systems present an opportunity to mitigate such bias. NLP algorithms for non-English languages, including optical character recognition (OCR) of languages not using a Latin alphabet, are required for such systems.</p></sec></sec><sec><title>Task 4: devise the search strategy</title><sec><title>Task description</title><p>Devising the search is distinct from conducting the search. The correct search strategy is critical to ensure that the review is not biased by the easily accessible studies. The search strategy describes what keywords will be used in searches, which databases will be searched [<xref ref-type="bibr" rid="B10">10</xref>], if and how citations will be tracked [<xref ref-type="bibr" rid="B46">46</xref>,<xref ref-type="bibr" rid="B47">47</xref>], and what and how evidence will be identified in non-database sources (such as reference checking and expert contacts) [<xref ref-type="bibr" rid="B48">48</xref>]. All Cochrane systematic review protocols, and many others, undergo peer review before the search is conducted.</p></sec><sec><title>Automation potential</title><p>Automatic search strategy creation by using the PICO of the planned question could form part of a decision support tool that ensures the strategy is consistent with the review question, includes both general and specific databases as needed, and that tested citation tracking methodologies are used.</p></sec><sec><title>Future research</title><p>Decision support tools for search strategy derivation could also suggest data sources, keywords (e.g., MeSH terms [<xref ref-type="bibr" rid="B49">49</xref>]) and search strategies [<xref ref-type="bibr" rid="B30">30</xref>,<xref ref-type="bibr" rid="B34">34</xref>,<xref ref-type="bibr" rid="B50">50</xref>] most suitable for the research question. It could also warn against search strategies that are too tailored to a particular database and that would thus be ineffective in others. This may involve understanding of the indexing terms (e.g., MeSH versus EMTREE) and structures of different databases (e.g., PubMed versus EMBASE).</p><p>Research into natural language processing is needed for algorithms that can understand a clinical question in a rich sense: extract its context, classify it into a type of clinical question, and/or identify pertinent keywords from the research question. An up-to-date database of clinical papers classified by type is needed to help decision support systems find, identify, and suggest the most suitable databases for the task.</p></sec></sec><sec><title>Task 5: search</title><sec><title>Task description</title><p>The biomedical literature is the main source of evidence for systematic reviews; however, some research studies can only be found outside of literature databases - the so called grey literature - and sometimes not at all [<xref ref-type="bibr" rid="B51">51</xref>,<xref ref-type="bibr" rid="B52">52</xref>]. For reviews to be systematic, the search task has to ensure all relevant literature is retrieved (perfect recall), even at the cost of retrieving up to tens of thousands of irrelevant documents [<xref ref-type="bibr" rid="B52">52</xref>] (precision as low as 0.3% [<xref ref-type="bibr" rid="B52">52</xref>]). It also implies that multiple databases have to be searched. Therefore, reviewers require specific knowledge of dozens of literary and non-literary databases, each with its own search engine, metadata, and vocabulary [<xref ref-type="bibr" rid="B53">53</xref>-<xref ref-type="bibr" rid="B55">55</xref>]. Interoperability among databases is rare [<xref ref-type="bibr" rid="B56">56</xref>]. Variation among search engines is large. Some have specialized query languages that may include logical operators such as OR, AND, and NOT, syntax for querying specific fields such as ‘authors’ and ‘year’, operators such as ADJ (adjacent to) and NEAR, and controlled keyword vocabularies such as Medical Subject Heading (MeSH) [<xref ref-type="bibr" rid="B49">49</xref>] for PubMed, and EMTREE for EMBASE.</p><p>Studies that investigated the efficacy of trial registration found that only about half of the clinical trials are published [<xref ref-type="bibr" rid="B57">57</xref>] and not all published trials are registered [<xref ref-type="bibr" rid="B58">58</xref>]. This indicates that registries and the literature should both be searched.</p><p>The most commonly used databases in systematic reviews are the general databases MEDLINE, Cochrane Library, and EMBASE [<xref ref-type="bibr" rid="B55">55</xref>] but hundreds of specialty databases also exists, e.g., CINAHL for nursing. The main non-literary sources are the American clinicaltrials.gov, the World Health Organization's International Clinical Trials Registration Platform [<xref ref-type="bibr" rid="B59">59</xref>], and the European clinicaltrialsregister.eu.</p><p>Studies that measured overlap between PubMed (which covers MEDLINE) and EMBASE found only a small number of articles that appeared in both [<xref ref-type="bibr" rid="B60">60</xref>-<xref ref-type="bibr" rid="B62">62</xref>]. These studies show why searching multiple databases is important. A study that measured overlap between Google Scholar and PubMed found Google Scholar to lead to the retrieval of twice as many relevant articles as well as twice as many irrelevant ones [<xref ref-type="bibr" rid="B63">63</xref>]. This study indicates that simply merging all databases into one large database improves recall without requiring searching multiple databases, but does not accelerate the systematic review.</p></sec><sec><title>Automation potential</title><p>Automation of the search task will reduce the time it takes to conduct the search and ensure that the translation of the generic search strategy into database-specific queries retains the integrity of the protocol. More comprehensive search (i.e., increased recall) will see to the inclusion of more evidence in the review. Better targeted search (i.e., increased precision) can reduce the number of papers that need critical appraisal in later stages.</p></sec><sec><title>Current systems</title><p>Decisions support for searching can utilize algorithms of at least one of the following: (i) algorithms that increase the utility (in terms of precision and recall) of user-entered queries and (ii) algorithms that help users write better search queries. Some web search engines already use a combination of both classes (e.g., Google).</p><p>Searching for clinical trials can be supported by specifically crafted filters and strategies [<xref ref-type="bibr" rid="B29">29</xref>-<xref ref-type="bibr" rid="B34">34</xref>]. Automatic Query Expansion (AQE) is the collective name for algorithms that modify the user's query before it is processed by the search engine [<xref ref-type="bibr" rid="B64">64</xref>]. Typically, AQE algorithms are implemented in the search engine itself but can also be provided by third parties. For example, HubMed (<ext-link ext-link-type="uri" xlink:href="http://www.HubMed.org">http://www.HubMed.org</ext-link>) provides alternative AQE algorithms unrelated to those offered by PubMed and without association with the National Library of Medicine which hosts MEDLINE. Examples of AQE algorithms include the following:</p><p>• Synonym expansion - automatically adding synonymous keywords to the search query to ensure that papers that use the synonym, and not the keyword used by the user, are retrieved in the search results;</p><p>• Word sense disambiguation - understanding keywords in the context of the search, replacing them with more suitable synonyms and dictionary definitions and removing unhelpful keywords; and</p><p>• Correct spelling mistakes and replace contractions, abbreviations, etc., with expanded terms more likely to be used in published work.</p><p>Document clustering algorithms group similar documents in a database. Search results are then computed by the database as document clusters that partially match the search query (i.e., only some of the clusters' members match the query) [<xref ref-type="bibr" rid="B65">65</xref>]. Clustering algorithms differ in the manner in which they identify documents as similar (i.e., by their similarity functions). Examples of document similarity measures include ones based on the terms that appear in the documents [<xref ref-type="bibr" rid="B66">66</xref>], and ones based on papers they cite [<xref ref-type="bibr" rid="B67">67</xref>].</p><p>A limited number of search engines help the user come up with better search queries. Google, and now many other web search engines, use keyword suggestion as soon as the user begins to type a search query, usually by suggesting terms that other users suggested previously. It may be more appropriate in the medical domain, to use a medical dictionary rather than previous searches, for example, to reduce spelling mistakes.</p><p>Meta-search engines are systems that amalgamate searches from several source databases. An automatic systematic review system that queries multiple databases would thus be a kind of meta-search engine by definition. An example of a basic type of meta-search engine is QUOSA (Quosa Inc. 2012). It lets the user query several source databases individually, and collects the results into a single list.</p><p>Federated search engines are meta-search engines that allow one to query multiple databases in parallel with a single query. Federated search engines need to translate the reviewer's query into the source databases' specific query languages and combine the results [<xref ref-type="bibr" rid="B68">68</xref>-<xref ref-type="bibr" rid="B70">70</xref>].</p><p>Quick Clinical [<xref ref-type="bibr" rid="B70">70</xref>] is a federated search engine with a specialized user interface for querying clinical evidence. The interface prompts the user for a query class, and then for specific fields suitable for the chosen query class (Figure <xref ref-type="fig" rid="F2">2</xref>). This interface not only prompts users to elaborate on their questions, it also guides inexpert users on how to ask clinical questions that will provide them with meaningful results [<xref ref-type="bibr" rid="B71">71</xref>]. Quick Clinical's AQE algorithm enriches keywords differently depending on which field they were typed in [<xref ref-type="bibr" rid="B70">70</xref>].</p><fig id="F2" position="float"><label>Figure 2</label><caption><p><bold>A screen capture of the Quick Clinical query screen from the smartphone app version.</bold> The Profile pull-down menu lets one select the class of question being asked (e.g. medication, diagnosis, patient education). The query fields are chosen to suit the question class. The four query fields shown (Disease, Drug, Symptom and Other) are taken from the Therapy question class.</p></caption><graphic xlink:href="2046-4053-3-74-2"></graphic></fig><p>The Turning Research Into Practice (TRIP) [<xref ref-type="bibr" rid="B72">72</xref>] database uses multiple query fields including to search PubMed. The query fields for population, intervention, comparison, and outcome (PICO) to enrich the query with terms that improve precision. In this way, TRIP uses knowledge about evidence-based medicine to improve evidence searches. TRIP is designed to provide quick access to evidence in lieu of a systematic review and its usefulness to systematic reviewers is yet to be demonstrated.</p></sec><sec><title>Future research</title><p>Expert searchers use iterative refinement of their search queries by a quick examination of the search results. As opposed to federated search engines which, in parallel, query each database once, expert searchers use a strategy of a series of queries to one database at a time. The automation of such a sequential searching is not yet explored.</p><p>The automatic translation of search queries between literature databases is not trivial. While general technical approaches exist and can convert the standard Boolean operators [<xref ref-type="bibr" rid="B70">70</xref>], these have yet to be demonstrated on translations that include database-specific vocabularies such as MeSH and EMTREE. BioPortal [<xref ref-type="bibr" rid="B73">73</xref>] is an ontology that maps vocabularies and includes MeSH but still lacks mappings to EMTREE or other literary database vocabularies.</p><p>A recent push to register all trials [<xref ref-type="bibr" rid="B74">74</xref>] has led to the creation of trial registries by governments [<xref ref-type="bibr" rid="B75">75</xref>-<xref ref-type="bibr" rid="B77">77</xref>] and health organizations [<xref ref-type="bibr" rid="B78">78</xref>,<xref ref-type="bibr" rid="B79">79</xref>]. The feasibility for future evidence search engines making use of such registries to find linked trial results and/or contact information for trial administrators, should yet be investigated.</p></sec></sec><sec><title>Task 6: de-duplicate</title><sec><title>Task description</title><p>Whenever citations are obtained from multiple sources, combining the results requires removing duplicate citations from the result lists [<xref ref-type="bibr" rid="B80">80</xref>]. Challenges in citation de-duplication arise due to variation in indexed metadata (e.g., DOI, ISBN, and page numbers are not always included), misspelling (e.g., in article title, journal name), abbreviation formats and legitimately duplicate information (e.g., identical author lists, different authors with the same name).</p><p>If the same study has more than one report - possibly with different author lists, different titles, and in different journals - both papers should often be cited, but they should only be included in meta-analysis as one trial [<xref ref-type="bibr" rid="B81">81</xref>]. Citation information alone cannot resolve this type of duplication which is also called ‘studification’. Characteristics needed to detect duplicated trials are only present in the text of the articles. This type of de-duplication is called study de-duplications as it aims to identify two distinct reports of the same study.</p></sec><sec><title>Automation potential</title><p>Both types of de-duplication are largely manual and time-consuming tasks and their automation has the potential to save many days that clinical librarians routinely spent on them. Study de-duplication, although the rarer of the two, is only usually detected after data have been extracted from both papers, after the authors have been contacted or sometimes not at all. Automatic de-duplication of the study can thus save precious time and free reviewers and clinical librarians to pursue other tasks.</p></sec><sec><title>Current systems</title><p>Current technologies focus on citation de-duplication [<xref ref-type="bibr" rid="B80">80</xref>]. Many citation managers (e.g., EndNote [<xref ref-type="bibr" rid="B82">82</xref>], ProCite [<xref ref-type="bibr" rid="B83">83</xref>]) already have automatic search for duplicate records to semi-automate citation de-duplication quite accurately [<xref ref-type="bibr" rid="B78">78</xref>,<xref ref-type="bibr" rid="B79">79</xref>]. Heuristic [<xref ref-type="bibr" rid="B84">84</xref>], machine learning (ML) algorithms and probabilistic string matching are used to compare author names, journal, conference or book names (known as venue) and titles of citations. This task can thus depend on our ability to accurately extract field information from citation strings which is also subject to ongoing research [<xref ref-type="bibr" rid="B85">85</xref>-<xref ref-type="bibr" rid="B89">89</xref>].</p></sec><sec><title>Future research</title><p>Future research should focus on study de-duplication which is distinct from citation de-duplication. Study de-duplication is trivial after information is extracted from the trial paper; however, the aim is to prevent unnecessary information extraction. Study de-duplication could be the result of studies being presented in different ways, in different formats or at different stages of the study and therefore requires sophisticated NLP algorithms that can infer a relationship between such papers. Many plagiarism detection algorithms have mechanisms to understand paper structure and find similarities between papers [<xref ref-type="bibr" rid="B90">90</xref>-<xref ref-type="bibr" rid="B92">92</xref>]. Such algorithms may guide development of algorithms that extract relationships between trial papers. Trial registries with multiple associated papers, and databases of extracted trial data could be used for automatic study de-duplication. Databases of manually extracted trial data are already maintained by some Cochrane group editors for specific domains but are not yet available for exploitation by automatic systems.</p></sec></sec><sec><title>Task 7: screen abstracts</title><sec><title>Task description</title><p>As the main aim of the retrieval task is to retrieve all relevant literature (perfect recall), the aim of the appraisal tasks is to exclude all irrelevant literature inadvertently retrieved with it. When appraising scientific papers' relevance to a systematic review, the vast majority of documents are excluded [<xref ref-type="bibr" rid="B10">10</xref>]. Exclusion is based on the relevance of the evidence to the research question and often an evaluation of the risk of bias in the trial.</p><p>Appraisal is normally conducted in two phases. First, titles and abstracts are used to quickly screen citations to exclude the thousands of papers that do not report trials or are trials not in the target population, intervention or do not measure the right outcome. This saves the reviewers from the lengthy process of retrieving the full text of many irrelevant papers.</p></sec><sec><title>Automation potential</title><p>Despite the apparent brevity of the screening of title and abstract, due to the large number to be screened, this is an error-prone and time-consuming task. Decision support systems can highlight information for reviewers to simplify the cognitive requirements of the task and save time.</p><p>A second reviewer is usually needed to screen the same abstracts. The two reviewers may meet to resolve any disagreements. Automatic screening systems could thus be used as a tie-breaker to resolve disagreements, and/or replace one or both screeners.</p></sec><sec><title>Current systems</title><p>Decision support systems use NLP to automatically highlight sentences and phrases that reviewers are likely to use for appraisal [<xref ref-type="bibr" rid="B93">93</xref>,<xref ref-type="bibr" rid="B94">94</xref>]. Often, information highlighting algorithms focus on PICO elements reported in the trial [<xref ref-type="bibr" rid="B95">95</xref>-<xref ref-type="bibr" rid="B97">97</xref>] but may also focus on the size of the trial [<xref ref-type="bibr" rid="B97">97</xref>,<xref ref-type="bibr" rid="B98">98</xref>], and its randomization procedure [<xref ref-type="bibr" rid="B95">95</xref>,<xref ref-type="bibr" rid="B97">97</xref>].</p><p>Existing commercial decision support systems are focused on helping reviewers manage inclusion and exclusion decisions, flag disagreements between reviewers and resolve them sometimes automatically [<xref ref-type="bibr" rid="B99">99</xref>-<xref ref-type="bibr" rid="B101">101</xref>].</p><p>Automation approaches are to use ML to infer exclusion and inclusion rules by observing a human screener [<xref ref-type="bibr" rid="B102">102</xref>-<xref ref-type="bibr" rid="B106">106</xref>], a class of algorithms called supervised ML. The Abstrackr system [<xref ref-type="bibr" rid="B103">103</xref>,<xref ref-type="bibr" rid="B104">104</xref>] observes inclusion and exclusion decisions made by a human reviewer. The system then extracts keywords from the abstracts and titles of the screened documents and builds models that mimic the user's decisions. When enough included as well as excluded documents are observed by the system, Abstrackr can automatically continue to screen the remaining reports and thus halve the total workload that the human reviewer would otherwise need to bear.</p><p>Current appraisal systems can reduce the number of documents that need to be appraised manually by between 30% and 50%, usually at the cost of up to 5% reduction in recall [<xref ref-type="bibr" rid="B102">102</xref>-<xref ref-type="bibr" rid="B106">106</xref>]. Another demonstration of an automatic screening algorithm alerts about new evidence that should trigger a systematic review update [<xref ref-type="bibr" rid="B106">106</xref>]. That system, however, did not achieve reliably high precision and recall.</p></sec><sec><title>Future research</title><p>A simple approach is to use the document type associated with each document in PubMed to remove all documents not classified as clinical trials. However, this approach is not reliable because it only works for PubMed and document labeled as randomized controlled trials are not necessarily reports of trial outcomes but could also be trial protocols, reviews of clinical trials, etc. A more accurate approach is therefore to use document classes together with specific keywords [<xref ref-type="bibr" rid="B107">107</xref>]. An alternative approach, not yet tested in this domain, is to get experts to develop rule-based systems using knowledge-acquisition systems.</p><p>Heuristic and ML systems can be complementary, and combinations of these approaches have proven accurate in other domains [<xref ref-type="bibr" rid="B108">108</xref>]. In future automated systematic reviews, we are likely to see a combination of rule-based and machine-learning algorithms.</p></sec></sec><sec><title>Task 8: obtain full text articles</title><sec><title>Task description</title><p>Obtaining the full text of published articles is technical, tedious and resource demanding. In the current process (Figure <xref ref-type="fig" rid="F1">1</xref>), full-text fetching is minimized by breaking the screening task into two parts. Known obstacles to full text fetching are a lack of standardized access to literature, burdensome subscription models and limited archival and electronic access. Full text retrieval requires a navigation through a complex network of links that spans multiple websites, paper-based content and even email. The problems are more pronounced for older studies.</p></sec><sec><title>Automation potential</title><p>Automation of full text retrieval has the potential to drastically change the systematic review process. Some tasks, such as screening from abstracts, would become redundant as screening of full text is superior. Other tasks, such as de-duplication, would be simplified as there would not be a need to rely on meta-data. Citation tracking (see below) may lead to systems that retrieve the required evidence primarily through means other than search.</p></sec><sec><title>Current systems</title><p>OvidSP [<xref ref-type="bibr" rid="B109">109</xref>] and PubMed Central [<xref ref-type="bibr" rid="B110">110</xref>] provide unified access to journal articles from multiple databases and journals and thus support full text retrieval. Some databases such as Google Scholar [<xref ref-type="bibr" rid="B111">111</xref>] and Microsoft Academic Search [<xref ref-type="bibr" rid="B112">112</xref>] provide multiple links for most documents that include the primary publisher's link, institutional copies and links to other databases, which may offer additional access options. Some reference managers such as EndNote [<xref ref-type="bibr" rid="B82">82</xref>] and ProCite [<xref ref-type="bibr" rid="B83">83</xref>] have full text retrieval functions.</p></sec><sec><title>Future research</title><p>Research on intelligent software agents that can autonomously navigate from search results to full text articles along several possible routes is still needed. Such systems could also extract corresponding author name, if it is known, send e-mail requests, and extract attachments from replies. Trial registries already contain author information and may contain links to full text papers could be used for this purpose.</p></sec></sec><sec><title>Task 9: screen the full text articles</title><sec><title>Task description</title><p>In the second stage of appraisal, reviewers screen using full text describing the trials not previously excluded. In this phase, a more detailed look at the trial allows the reviewer to exclude or include trials by inspecting subtle details not found in the abstract. This task is different from screening using abstracts in that a more careful understanding of the trial is needed. Pertinent information may be reported in sentences that have to be read in context.</p></sec><sec><title>Automation potential</title><p>Despite the smaller number of trials to screen, each trial requires more time and attention. A second reviewer is often used, and agreement by consensus is sought which make the process even more resource and time demanding. Computational systems could automatically resolve disagreements or replace one or both reviewers to reduce resource demand.</p></sec><sec><title>Current systems</title><p>Decision support systems that screen abstracts may work on full-text screening unmodified. Elements that are not normally present in the abstract, such as figures and tables, and that contain information pertinent to appraisal, are not yet mined by current systems. Systems that do make use of such information have been proposed [<xref ref-type="bibr" rid="B94">94</xref>,<xref ref-type="bibr" rid="B113">113</xref>] but have not yet seen adoption in this domain.</p></sec><sec><title>Future research</title><p>Research is required to evaluate such systems for their reliability, and integrate them into systematic review support systems.</p></sec></sec><sec><title>Task 10: ‘snowballing’ by forward and backward citation search</title><sec><title>Task description</title><p>Limitations of keyword searching have led to the development of secondary analysis of search results [<xref ref-type="bibr" rid="B114">114</xref>]. One such analysis method is called snowballing [<xref ref-type="bibr" rid="B46">46</xref>], which recursively pursues relevant references cited in retrieved papers (also called citation tracking) and adding them to the search results. While this is a time-consuming step, it has been shown to improve retrieval [<xref ref-type="bibr" rid="B115">115</xref>], and is a recommended practice [<xref ref-type="bibr" rid="B46">46</xref>]. Unlike keywords search, snowballing does not require specific search terms but allows the accumulation of multiple searches from different publishing authors [<xref ref-type="bibr" rid="B47">47</xref>].</p></sec><sec><title>Automation potential</title><p>Automatic snowballing has the potential to increase recall by following more citations than a human reviewer would. A case study on forward citation tracking on depression and coronary heart disease has shown to identify more eligible articles and reduce bias [<xref ref-type="bibr" rid="B47">47</xref>]. A review on checking reference lists to find additional studies for systematic reviews also found that citation tracking identify additional studies [<xref ref-type="bibr" rid="B115">115</xref>].</p><p>When combined with automatic appraisal and document fetching, automatic snowballing can have a compound effect by iteratively following select citations until no new relevant citations are found. A recent study [<xref ref-type="bibr" rid="B116">116</xref>] examined the potential for using citation networks for document retrieval. That study shows that in most reviews tested, forward and backward citation tracking will lead to all relevant trials manually identified by systematic reviewers. Snowballing would benefit almost all searches (>95%) but does not completely replace database searches based on search terms.</p></sec><sec><title>Current systems</title><p>Automatic citation extraction tools [<xref ref-type="bibr" rid="B85">85</xref>-<xref ref-type="bibr" rid="B89">89</xref>] are core to snowballing systems. Some databases such as Web of Science and Microsoft Academic Search provide citation networks (for a fee) that can be traversed to find related literature. Those networks are limited to papers indexed in the same database.</p></sec><sec><title>Future research</title><p>The risk of an automatic snowballing system is that it can get out of control, retrieving more articles than it is feasible to appraise manually. Formal definitions of robust stopping criteria are therefore pivotal research in this area. Integration with automatic appraisal and fetching systems will help to improve the practical application of snowballing systems.</p></sec></sec><sec><title>Task 11: extract data</title><sec><title>Task description</title><p>Data extraction is the identification of trial features, methods, and outcomes in the text of relevant trials. In many trial papers, primary information is only published in graphical form and primary data needs to be extracted from these plots as accurately as resolution permits. Usually two reviewers perform the task independently and resolve disagreements by consensus. Extracting data from text is one of the most time-consuming tasks of the systematic review.</p></sec><sec><title>Automation potential</title><p>A great deal of expertise, time, and resources can be saved through partial and complete automation of this time-consuming task. As two reviewers perform it in parallel, a computer extractor could serve as an arbiter on disagreements, or replace one or both reviewers. The identification, extraction of the number of patients in each arm, and extraction and association of outcome quantities with the same arms, can be challenging even to experienced reviewers. The information may be present in several sections of the trial paper making the extraction a complex cognitive task. Even partial automation may reduce the expertise required to complete this task, reduce errors, and save time.</p></sec><sec><title>Current systems</title><p>The approach currently used to automate extraction of information from clinical trial papers has two stages. The first sub-task is to reduce the amount of text to be processed using information-highlighting algorithms [<xref ref-type="bibr" rid="B95">95</xref>-<xref ref-type="bibr" rid="B97">97</xref>]. The second sub-task is to associate extracted elements with experimental arms and with outcomes [<xref ref-type="bibr" rid="B117">117</xref>-<xref ref-type="bibr" rid="B119">119</xref>].</p><p>ExaCT is an information-highlighting algorithm [<xref ref-type="bibr" rid="B95">95</xref>]. It classifies sentences and sometimes phrases that contain about 20 elements used in information extraction and screening. The classes include PICO, randomization, and subclasses of elements. For example, subclasses of intervention elements include route of treatment, frequency of treatment, and duration of treatment. ExaCT can often distinguish between sentences that describe the intervention and from those that describe the control but does not associate, e.g., measured outcomes with each arm.</p><p>To identify intervention sentences, a highlighting algorithm may be used to identify drug names in the title of the document. A template [<xref ref-type="bibr" rid="B118">118</xref>] or a statistical model [<xref ref-type="bibr" rid="B119">119</xref>] is then used to associate the number of patients in a particular arm in the study. Each outcome in each arm can then be summarized by two numbers: number of outcome events in the study arm and number of patients in the same study arm. Extracting all outcomes for all arms of the trial then gives a structured summary of the trial.</p><p>Software tools for digitizing graphs use line recognition algorithms to help a reviewer trace plots and extract raw data from them. Graph digitization software are not specific to systematic reviews and thus support the more common graph types such as X-Y and polar plots [<xref ref-type="bibr" rid="B120">120</xref>-<xref ref-type="bibr" rid="B122">122</xref>], but not survival curves, which are more common in clinical trials.</p></sec><sec><title>Future research</title><p>Current methods are still limited to processing one sentence at a time which severely limits the number of trials they can process. Future research systems will be required to create of summaries of trials based on information extracted from multiple sentences in different sections of the trial full text. For example, the trial may specify the number of patients randomized to each arm in one sentence (e.g., ‘<italic>Five hundred and seventy-seven (377 Japanese, 200 Chinese) patients treated with antihypertensive therapy (73.5% [n = 424] received concomitant ACEI), were given either once-daily olmesartan (10-40 mg) (n = 288) or placebo (n = 289) over 3.2 ± 0.6 years (mean ± SD).’</italic>) [<xref ref-type="bibr" rid="B123">123</xref>] and the number of events in each arm in another (‘<italic>In the olmesartan group, 116 developed the primary outcome (41.1%) compared with 129 (45.4%) in the placebo group (HR 0.97, 95% CI 0.75, 1.24; p = 0.791)’</italic>) [<xref ref-type="bibr" rid="B123">123</xref>]. In another example, the primary outcome is specified in the methods section (e.g., ‘duration of the first stage of labor’) [<xref ref-type="bibr" rid="B124">124</xref>] and only a reference to it can be found in the results section (e.g., ‘the first stage’) [<xref ref-type="bibr" rid="B124">124</xref>]. New NLP methods are required to be able to computationally reason about data from such texts.</p><p>Current methods all assume that trials can be modeled using a simple branching model in which patients are either assigned to one group or another or drop out [<xref ref-type="bibr" rid="B125">125</xref>]. While this model is suitable for a large proportion of published trials, other study designs have not yet been addressed.</p><p>As extraction methods develop, they will increasingly understand more study designs such as cross-over studies. Computationally reasoning over trials with a different number of arms and which are not controlled against placebo or ‘standard care’ would require significantly more elaborate computational reasoning [<xref ref-type="bibr" rid="B42">42</xref>].</p><p>Specific tools to extract survival curves will be beneficial to systematic review research. Research is also required to use optical character recognition (OCR) to digitize text (e.g., axes labels, data labels, plot legends) and combine the information with data extracted from the plot.</p><p>Databases of manually extracted trial data are already maintained by some Cochrane group editors for specific domains and collected in CRS [<xref ref-type="bibr" rid="B126">126</xref>]. Automatic extraction can populate such databases [<xref ref-type="bibr" rid="B43">43</xref>]. The feasibility of using trial databases in automatic systems is yet to be tested.</p></sec></sec><sec><title>Task 12: convert and synthesize data</title><sec><title>Task description</title><p>Synthesis may first require the conversion of numerical results to a common format, e.g., a common scale or a standardized mean difference, so that trials can be compared. Normally, comparison of continuous distributions is done according to the average and standard deviation (SD). Comparison of discrete measures uses relative risk (RR) and later converted to number needed to treat (NNT).</p></sec><sec><title>Automation potential</title><p>Conversion between statistical formats requires specific expertise in statistics. Automation of this task can thus reduce the requirement for specific training in statistics and to reduce the chance of the wrong conversion being used, e.g., confusing standard error and standard deviations.</p></sec><sec><title>Current systems</title><p>It is standard practice to use statistical packages for synthesis. However, much of the practice is still manual, time consuming, and error prone. We have identified only one study that attempted automatic extraction and synthesis of discrete distributions to NTT and RR formats [<xref ref-type="bibr" rid="B118">118</xref>].</p></sec><sec><title>Future research</title><p>The automatic synthesis system will have to automatically recognize the outcome format (e.g., as a confidence interval, average effect size, <italic>p</italic> value, etc.), select the statistical conversion, and translate the outcome to its average and standard deviation. These formulae may require extracted entities that may not be reported as part of the outcome, such as the number of patients randomized to that arm.</p></sec></sec><sec><title>Task 13: re-check the literature</title><sec><title>Task description</title><p>Due to the time lag between the initial search and the actual review (typically between 12 and 24 months), a secondary search may be conducted. The secondary search is designed to find trials indexed since the initial search. Often this task implies a repetition of many previous tasks such as appraisal, extraction, and synthesis making it potentially very expensive.</p></sec><sec><title>Automation potential</title><p>In an automatic review system, retrieval, appraisal, extraction, and synthesis would occur in quick succession and in a short amount of time. This would make this task altogether unnecessary. Consequentially, an automatic review process will have the compounded effect that will save much manual labor.</p></sec></sec><sec><title>Task 14: meta-analyze</title><sec><title>Task description</title><p>For a systematic review to be useful to a clinician, it needs to be presented in a clear, precise and informative manner [<xref ref-type="bibr" rid="B127">127</xref>,<xref ref-type="bibr" rid="B128">128</xref>]. Systematic reviews are often followed by a meta-analysis of the included trial results. The most common way to summarize a meta-analysis is using a forest plot [<xref ref-type="bibr" rid="B129">129</xref>] but other plots exist, such as Galbraith diagrams (also called radial diagrams) and L'Abbé plots. Other plots are used for data checking purposes, e.g., funnel plots [<xref ref-type="bibr" rid="B130">130</xref>] show whether publication bias is likely to affect the conclusion of the systematic review.</p></sec><sec><title>Automation potential</title><p>Transferring quantities between software packages is time consuming and error prone. Thus, integration of synthesis and graphing software has the potential to save time and reduce errors.</p></sec><sec><title>Current systems</title><p>Much of the meta-analysis is already automatic as software for combining, comparing, and reporting the meta-analysis graphically, are already in wide use [<xref ref-type="bibr" rid="B131">131</xref>-<xref ref-type="bibr" rid="B134">134</xref>].</p></sec><sec><title>Future research</title><p>Plotting is routinely manual with the aid of specialized software. Software for systematic reviews are included in specific software packages including Review Manager (RevMan) [<xref ref-type="bibr" rid="B40">40</xref>], Meta-Analyst [<xref ref-type="bibr" rid="B131">131</xref>], MetaDiSc [<xref ref-type="bibr" rid="B132">132</xref>], and MetaWin [<xref ref-type="bibr" rid="B133">133</xref>], all of which already produce Forest and other plots.</p></sec></sec><sec><title>Task 15: write up the review</title><sec><title>Task description</title><p>Write-up of a scientific paper according to reporting standards [<xref ref-type="bibr" rid="B135">135</xref>] is the major dissemination channel for systematic reviews. This process is prolonged and subject to review, editing, and publication delays [<xref ref-type="bibr" rid="B4">4</xref>,<xref ref-type="bibr" rid="B7">7</xref>]. It involves many menial sub-tasks that can be automated, such as typesetting figures and laying out the text. Other sub-tasks, such as interpreting the results and writing the conclusion, may require more manual interventions.</p></sec><sec><title>Automation potential</title><p>The automatic creation of a report from the protocol can save years in the production of each systematic review. Rather than the protocol and the written-up report requiring separate peer-review, only the protocol would be reviewed as the report will be produced directly from it. This will reduce the load on peer reviews and reduce delays further.</p></sec><sec><title>Current systems</title><p>The Cochrane Collaboration provides templates for the production of the systematic review protocol and text. These help reviewers ensure a complete and consistent final product. RevMan-HAL [<xref ref-type="bibr" rid="B136">136</xref>] is a program that provides templates and rules that generate the results and conclusion sections of Cochrane reviews from RevMan files that contain a meta-analysis. After the template is filled in, the reviewer may edit the final report.</p><p>A well-written report includes graphical and textual summaries. In addition to the forest plots and the other meta-analysis representations mentioned above, graphical summaries include tables and PRISMA flow diagrams. Microsoft Office Excel is often used to enter data into tables and copy them into the report. Tools exist for creating relevant diagrams including forest plots and PRISMA flow diagram [<xref ref-type="bibr" rid="B137">137</xref>].</p></sec><sec><title>Future research</title><p>Natural Language Generation (NLG) technology [<xref ref-type="bibr" rid="B138">138</xref>] can be used to write specific paragraphs within the review such as descriptions of the types of documents found, results of the appraisal and summaries of the findings. Disparity in data formats among tools mean that errors may be introduced when manually transferring data between tools, or in format incompatibilities. Better tool integration is needed.</p><p>Interpretation of meta-analysis results is not only done by the systematic reviewer. Clinicians and other users of the review may interpret the results and use their own interpretation.</p></sec></sec></sec><sec><title>Systems approach to systematic review automation</title><p>A ‘living’ systematic review is updated at the time it is required to be used in practice [<xref ref-type="bibr" rid="B139">139</xref>]. It is no longer a document that summarizes the evidence in the past. Rather, it is designed and tested once, and is run at a push of a button to execute a series of search, appraisal, information extraction, summarization, and report generation algorithms on all the data available. Like other computer applications, the systematic review protocol itself can be modified, updated, and corrected and then redistributed so that such amendments are reflected in new systematic reviews based on the protocol.</p><p>Much research on integrated automatic review systems is still needed. Examples of complex systems that have evolved as combinations of simpler ones are abundant and provide hope as well as valuable components in a future system. In this survey, we found that systems designed to perform many of the systematic review tasks are already in wide use, in development, or in research. Semi-automated decision support systems will advance the end goal of completely autonomous systematic review systems.</p><p>An enterprise to automate systematic review has many stakeholders including clinicians, journal editors, registries, clinical scientists, reviewers, and informaticians. Collaboration between the stakeholders may include publication requirements of registration of reviews by medical editor committees, better adherence to CONSORT and PRISMA statements, and explicit PICO reporting. Accommodation in systematic review registries such as PROSPERO and standard data exchanges such as HL7 [<xref ref-type="bibr" rid="B140">140</xref>] may assist in the standardization and quality assurance of automatically generated systematic reviews.</p><p>The path to a fully automated systematic review system will continue to deliver a range of software applications that will benefit systematic reviewers directly as well as some that will benefit the community less directly. For example, imperfect applications that can extract data only from CONSORT-compliant abstracts can help editors conduct pre-submission quality assurance. Software that automatically differentiates citations to published papers from grey literature can also be used to create a new registry for grey literature. With small modifications, search strategies from previous systematic reviews can be used by guideline developers to find evidence summaries and identify gaps in knowledge.</p></sec></sec><sec sec-type="conclusions"><title>Conclusion</title><p>Synthesis for evidence based medicine is quickly becoming unfeasible because of the exponential growth in evidence production. Limited resources can be better utilized with computational assistance and automation to dramatically improve the process. Health informaticians are uniquely positioned to take the lead on the endeavor to transform evidence-based medicine through lessons learned from systems and software engineering. The limited resources dedicated to evidence synthesis can be better utilized with computational assistance and automation.</p><p>Conducting systematic reviews faster, with fewer resources, will produce more reviews to answer more clinical questions, keep them up to date, and require less training. Together, advances in the automation of systematic reviews will provide clinicians with more evidence-based answers and thus allow them to provide higher quality care.</p></sec><sec><title>Abbreviations</title><p>AQE: automatic (search) query expansion; CI: confidence interval; CINAHL: Cumulative Index to Nursing and Allied Health Literature; CONSORT: consolidated standards of reporting trials; CRS: Cochrane Register of Studies; DARE: Database of Abstracts of Reviews of Effects; DOI: digital object identifier; HL7: Health Level 7; HR: hazard ratio; ISBN: International Standard Book Number; MeSH: Medical Subject Headings; ML: machine learning; NLG: natural language generation; NLP: natural language processing; NNT: number needed to treat; OCR: optical character recognition; PICO: Population (or Problem): Intervention: Control and Outcome; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; PROSPERO: International Register of Prospective Systematic Review; RevMan: Review Manager; RR: relative risk; SD: standard deviation; TRIP: Turning Research Into Practice.</p></sec><sec><title>Competing interests</title><p>The authors declare that they have no competing interests.</p></sec><sec><title>Authors' contributions</title><p>GT designed and drafted the manuscript and conducted literature searches. PG provided domain guidance on systematic review tasks and processes. MKC conducted literature searches. AD provided domain expertise on clinical evidence and appraisal. FG conducted literature searches. EC provided domain guidance on decision support systems and informatics. All authors read and approved the final manuscript.</p></sec></body><back><sec><title>Acknowledgements</title><p>This work was funded by the National Health and Medical Research Centre for Research Excellence in eHealth (APP1032664).</p></sec><ref-list><ref id="B1"><mixed-citation publication-type="book"><name><surname>Sackett</surname><given-names>DL</given-names></name><name><surname>Straus</surname><given-names>S</given-names></name><name><surname>Richardson</surname><given-names>WS</given-names></name><name><surname>Rosenberg</surname><given-names>W</given-names></name><name><surname>Haynes</surname><given-names>RB</given-names></name><source>Evidence-based Medicine: How to Teach and Practice EBM</source><year>2000</year><publisher-name>Edinburgh: Churchill Livingstone</publisher-name></mixed-citation></ref><ref id="B2"><mixed-citation publication-type="book"><name><surname>Cochrane</surname><given-names>AL</given-names></name><article-title>1931-1971: a critical review, with particular reference to the medical profession</article-title><source>Medicines for the Year 2000</source><year>1979</year><publisher-name>London: Office of Health Economics</publisher-name><fpage>1</fpage><lpage>11</lpage></mixed-citation></ref><ref id="B3"><mixed-citation publication-type="journal"><name><surname>Jaidee</surname><given-names>W</given-names></name><name><surname>Moher</surname><given-names>D</given-names></name><name><surname>Laopaiboon</surname><given-names>M</given-names></name><article-title>Time to update and quantitative changes in the results of cochrane pregnancy and childbirth reviews</article-title><source>PLoS One</source><year>2010</year><volume>5</volume><fpage>e11553</fpage><pub-id pub-id-type="pmid">20644625</pub-id></mixed-citation></ref><ref id="B4"><mixed-citation publication-type="journal"><name><surname>Bastian</surname><given-names>H</given-names></name><name><surname>Glasziou</surname><given-names>P</given-names></name><name><surname>Chalmers</surname><given-names>I</given-names></name><article-title>Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?</article-title><source>PLoS Med</source><year>2010</year><volume>7</volume><fpage>e1000326</fpage><pub-id pub-id-type="pmid">20877712</pub-id></mixed-citation></ref><ref id="B5"><mixed-citation publication-type="journal"><name><surname>Tsafnat</surname><given-names>G</given-names></name><name><surname>Dunn</surname><given-names>A</given-names></name><name><surname>Glasziou</surname><given-names>P</given-names></name><name><surname>Coiera</surname><given-names>E</given-names></name><article-title>The automation of systematic reviews</article-title><source>BMJ</source><year>2013</year><volume>346</volume><fpage>f139</fpage><pub-id pub-id-type="pmid">23305843</pub-id></mixed-citation></ref><ref id="B6"><mixed-citation publication-type="journal"><name><surname>Smith</surname><given-names>R</given-names></name><name><surname>Chalmers</surname><given-names>I</given-names></name><article-title>Britain’s gift: a “Medline” of synthesised evidence</article-title><source>BMJ</source><year>2001</year><volume>323</volume><fpage>1437</fpage><lpage>1438</lpage><pub-id pub-id-type="pmid">11751342</pub-id></mixed-citation></ref><ref id="B7"><mixed-citation publication-type="journal"><name><surname>Shojania</surname><given-names>KG</given-names></name><name><surname>Sampson</surname><given-names>M</given-names></name><name><surname>Ansari</surname><given-names>MT</given-names></name><name><surname>Ji</surname><given-names>J</given-names></name><name><surname>Doucette</surname><given-names>S</given-names></name><name><surname>Moher</surname><given-names>D</given-names></name><article-title>How quickly do systematic reviews go out of date? A survival analysis</article-title><source>Ann Intern Med</source><year>2007</year><volume>147</volume><fpage>224</fpage><lpage>233</lpage><pub-id pub-id-type="pmid">17638714</pub-id></mixed-citation></ref><ref id="B8"><mixed-citation publication-type="journal"><name><surname>Takwoingi</surname><given-names>Y</given-names></name><name><surname>Hopewell</surname><given-names>S</given-names></name><name><surname>Tovey</surname><given-names>D</given-names></name><name><surname>Sutton</surname><given-names>AJ</given-names></name><article-title>A multicomponent decision tool for prioritising the updating of systematic reviews</article-title><source>BMJ</source><year>2013</year><volume>347</volume><fpage>f7191</fpage><pub-id pub-id-type="pmid">24336453</pub-id></mixed-citation></ref><ref id="B9"><mixed-citation publication-type="journal"><name><surname>Adams</surname><given-names>CE</given-names></name><name><surname>Polzmacher</surname><given-names>S</given-names></name><name><surname>Wolff</surname><given-names>A</given-names></name><article-title>Systematic reviews: work that needs to be done and not to be done</article-title><source>J Evid-Based Med</source><year>2013</year><volume>6</volume><fpage>232</fpage><lpage>235</lpage><pub-id pub-id-type="pmid">24325416</pub-id></mixed-citation></ref><ref id="B10"><mixed-citation publication-type="book"><collab>The Cochrane Collaboration</collab><source>Cochrane Handbook for Systematic Reviews of Interventions</source><year>2011</year><edition>51</edition><comment><ext-link ext-link-type="uri" xlink:href="http://www.cochrane.org/training/cochrane-handbook">http://www.cochrane.org/training/cochrane-handbook</ext-link></comment></mixed-citation></ref><ref id="B11"><mixed-citation publication-type="book"><name><surname>Glasziou</surname><given-names>P</given-names></name><name><surname>Beller</surname><given-names>E</given-names></name><name><surname>Thorning</surname><given-names>S</given-names></name><name><surname>Vermeulen</surname><given-names>M</given-names></name><source>Manual for systematic reviews</source><year>2013</year><publisher-name>Gold Coast, Queensland, Australia: Centre for Research on Evidence Based Medicine</publisher-name></mixed-citation></ref><ref id="B12"><mixed-citation publication-type="journal"><name><surname>Rennels</surname><given-names>GD</given-names></name><name><surname>Shortliffe</surname><given-names>EH</given-names></name><name><surname>Stockdale</surname><given-names>FE</given-names></name><name><surname>Miller</surname><given-names>PL</given-names></name><article-title>A computational model of reasoning from the clinical literature</article-title><source>Comput Methods Programs Biomed</source><year>1987</year><volume>24</volume><fpage>139</fpage><lpage>149</lpage><pub-id pub-id-type="pmid">3297479</pub-id></mixed-citation></ref><ref id="B13"><mixed-citation publication-type="journal"><name><surname>Wallace</surname><given-names>BC</given-names></name><name><surname>Dahabreh</surname><given-names>IJ</given-names></name><name><surname>Schmid</surname><given-names>CH</given-names></name><name><surname>Lau</surname><given-names>J</given-names></name><name><surname>Trikalinos</surname><given-names>TA</given-names></name><article-title>Modernizing the systematic review process to inform comparative effectiveness: tools and methods</article-title><source>J Comp Effect Res</source><year>2013</year><volume>2</volume><fpage>273</fpage><lpage>282</lpage></mixed-citation></ref><ref id="B14"><mixed-citation publication-type="journal"><name><surname>Coiera</surname><given-names>E</given-names></name><article-title>Information epidemics, economics, and immunity on the internet</article-title><source>BMJ</source><year>1998</year><volume>317</volume><fpage>1469</fpage><lpage>1470</lpage><pub-id pub-id-type="pmid">9831570</pub-id></mixed-citation></ref><ref id="B15"><mixed-citation publication-type="other"><name><surname>Cohen</surname><given-names>AM</given-names></name><name><surname>Adams</surname><given-names>CE</given-names></name><name><surname>Davis</surname><given-names>JM</given-names></name><name><surname>Yu</surname><given-names>C</given-names></name><name><surname>Yu</surname><given-names>PS</given-names></name><name><surname>Meng</surname><given-names>W</given-names></name><name><surname>Duggan</surname><given-names>L</given-names></name><name><surname>McDonagh</surname><given-names>M</given-names></name><name><surname>Smalheiser</surname><given-names>NR</given-names></name><article-title>Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools</article-title><source>Proceedings of the 1st ACM International Health Informatics Symposium</source><year>2010</year><fpage>376</fpage><lpage>380</lpage></mixed-citation></ref><ref id="B16"><mixed-citation publication-type="journal"><name><surname>Counsell</surname><given-names>C</given-names></name><article-title>Formulating questions and locating primary studies for inclusion in systematic reviews</article-title><source>Ann Intern Med</source><year>1997</year><volume>127</volume><fpage>380</fpage><lpage>387</lpage><pub-id pub-id-type="pmid">9273830</pub-id></mixed-citation></ref><ref id="B17"><mixed-citation publication-type="journal"><name><surname>Oxman</surname><given-names>AD</given-names></name><name><surname>Sackett</surname><given-names>DL</given-names></name><name><surname>Guyatt</surname><given-names>GH</given-names></name><name><surname>Group</surname><given-names>E-BMW</given-names></name><article-title>Users’ guides to the medical literature: I. How to get started</article-title><source>JAMA</source><year>1993</year><volume>270</volume><fpage>2093</fpage><lpage>2095</lpage><pub-id pub-id-type="pmid">8411577</pub-id></mixed-citation></ref><ref id="B18"><mixed-citation publication-type="journal"><name><surname>Bragge</surname><given-names>P</given-names></name><name><surname>Clavisi</surname><given-names>O</given-names></name><name><surname>Turner</surname><given-names>T</given-names></name><name><surname>Tavender</surname><given-names>E</given-names></name><name><surname>Collie</surname><given-names>A</given-names></name><name><surname>Gruen</surname><given-names>RL</given-names></name><article-title>The Global Evidence Mapping Initiative: scoping research in broad topic areas</article-title><source>BMC Med Res Methodol</source><year>2011</year><volume>11</volume><fpage>92</fpage><pub-id pub-id-type="pmid">21682870</pub-id></mixed-citation></ref><ref id="B19"><mixed-citation publication-type="book"><name><surname>Snilstveit</surname><given-names>B</given-names></name><name><surname>Vojtkova</surname><given-names>M</given-names></name><name><surname>Bhavsar</surname><given-names>A</given-names></name><name><surname>Gaarder</surname><given-names>M</given-names></name><source>Evidence Gap Maps–A Tool for Promoting Evidence-Informed Policy and Prioritizing Future Research</source><year>2013</year><publisher-name>Washington, DC: The World Bank</publisher-name></mixed-citation></ref><ref id="B20"><mixed-citation publication-type="journal"><name><surname>Arksey</surname><given-names>H</given-names></name><name><surname>O’Malley</surname><given-names>L</given-names></name><article-title>Scoping studies: towards a methodological framework</article-title><source>Int J Soc Res Meth</source><year>2005</year><volume>8</volume><fpage>19</fpage><lpage>32</lpage></mixed-citation></ref><ref id="B21"><mixed-citation publication-type="journal"><name><surname>Bero</surname><given-names>L</given-names></name><name><surname>Busuttil</surname><given-names>G</given-names></name><name><surname>Farquhar</surname><given-names>C</given-names></name><name><surname>Koehlmoos</surname><given-names>TP</given-names></name><name><surname>Moher</surname><given-names>D</given-names></name><name><surname>Nylenna</surname><given-names>M</given-names></name><name><surname>Smith</surname><given-names>R</given-names></name><name><surname>Tovey</surname><given-names>D</given-names></name><article-title>Measuring the performance of the Cochrane library</article-title><source>Cochrane Database Syst Rev</source><year>2012</year><volume>12</volume><comment>ED000048</comment></mixed-citation></ref><ref id="B22"><mixed-citation publication-type="journal"><name><surname>Smalheiser</surname><given-names>NR</given-names></name><article-title>Informatics and hypothesis-driven research</article-title><source>EMBO Rep</source><year>2002</year><volume>3</volume><fpage>702</fpage><pub-id pub-id-type="pmid">12151321</pub-id></mixed-citation></ref><ref id="B23"><mixed-citation publication-type="journal"><name><surname>Malhotra</surname><given-names>A</given-names></name><name><surname>Younesi</surname><given-names>E</given-names></name><name><surname>Gurulingappa</surname><given-names>H</given-names></name><name><surname>Hofmann-Apitius</surname><given-names>M</given-names></name><article-title>‘HypothesisFinder:’ a strategy for the detection of speculative statements in scientific text</article-title><source>PLoS Comput Biol</source><year>2013</year><volume>9</volume><fpage>e1003117</fpage><pub-id pub-id-type="pmid">23935466</pub-id></mixed-citation></ref><ref id="B24"><mixed-citation publication-type="journal"><name><surname>Cohen</surname><given-names>T</given-names></name><name><surname>Whitfield</surname><given-names>GK</given-names></name><name><surname>Schvaneveldt</surname><given-names>RW</given-names></name><name><surname>Mukund</surname><given-names>K</given-names></name><name><surname>Rindflesch</surname><given-names>T</given-names></name><article-title>EpiphaNet: an interactive tool to support biomedical discoveries</article-title><source>J Biomed Disc Collab</source><year>2010</year><volume>5</volume><fpage>21</fpage></mixed-citation></ref><ref id="B25"><mixed-citation publication-type="journal"><name><surname>Murray</surname><given-names>CJ</given-names></name><article-title>Quantifying the burden of disease: the technical basis for disability-adjusted life years</article-title><source>Bull World Health Organ</source><year>1994</year><volume>72</volume><fpage>429</fpage><pub-id pub-id-type="pmid">8062401</pub-id></mixed-citation></ref><ref id="B26"><mixed-citation publication-type="journal"><name><surname>Chalmers</surname><given-names>I</given-names></name><name><surname>Glasziou</surname><given-names>P</given-names></name><article-title>Avoidable waste in the production and reporting of research evidence</article-title><source>Obst Gynecol</source><year>2009</year><volume>114</volume><fpage>1341</fpage><lpage>1345</lpage><pub-id pub-id-type="pmid">19935040</pub-id></mixed-citation></ref><ref id="B27"><mixed-citation publication-type="journal"><name><surname>Booth</surname><given-names>A</given-names></name><name><surname>Clarke</surname><given-names>M</given-names></name><name><surname>Ghersi</surname><given-names>D</given-names></name><name><surname>Moher</surname><given-names>D</given-names></name><name><surname>Petticrew</surname><given-names>M</given-names></name><name><surname>Stewart</surname><given-names>L</given-names></name><article-title>An international registry of systematic-review protocols</article-title><source>Lancet</source><year>2011</year><volume>377</volume><fpage>108</fpage><lpage>109</lpage><pub-id pub-id-type="pmid">20630580</pub-id></mixed-citation></ref><ref id="B28"><mixed-citation publication-type="book"><name><surname>Carini</surname><given-names>S</given-names></name><name><surname>Sim</surname><given-names>I</given-names></name><article-title>SysBank: a knowledge base for systematic reviews of randomized clinical trials</article-title><source>AMIA Annual Symposium Proceedings</source><year>2003</year><publisher-name>Washington, DC: American Medical Informatics Association</publisher-name><fpage>804</fpage></mixed-citation></ref><ref id="B29"><mixed-citation publication-type="journal"><name><surname>Glanville</surname><given-names>J</given-names></name><name><surname>Bayliss</surname><given-names>S</given-names></name><name><surname>Booth</surname><given-names>A</given-names></name><name><surname>Dundar</surname><given-names>Y</given-names></name><name><surname>Fernandes</surname><given-names>H</given-names></name><name><surname>Fleeman</surname><given-names>ND</given-names></name><name><surname>Foster</surname><given-names>L</given-names></name><name><surname>Fraser</surname><given-names>C</given-names></name><name><surname>Fry-Smith</surname><given-names>A</given-names></name><name><surname>Golder</surname><given-names>S</given-names></name><name><surname>Lefebvre</surname><given-names>C</given-names></name><name><surname>Miller</surname><given-names>C</given-names></name><name><surname>Paisley</surname><given-names>S</given-names></name><name><surname>Payne</surname><given-names>L</given-names></name><name><surname>Price</surname><given-names>A</given-names></name><name><surname>Welch</surname><given-names>K</given-names></name><article-title>So many filters, so little time: the development of a search filter appraisal checklist</article-title><source>J Med Libr Assoc</source><year>2008</year><volume>96</volume><fpage>356</fpage><lpage>361</lpage><pub-id pub-id-type="pmid">18974813</pub-id></mixed-citation></ref><ref id="B30"><mixed-citation publication-type="journal"><name><surname>Wong</surname><given-names>SS</given-names></name><name><surname>Wilczynski</surname><given-names>NL</given-names></name><name><surname>Haynes</surname><given-names>RB</given-names></name><article-title>Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE</article-title><source>Stud Health Technol Inform</source><year>2004</year><volume>107</volume><fpage>311</fpage><lpage>316</lpage><pub-id pub-id-type="pmid">15360825</pub-id></mixed-citation></ref><ref id="B31"><mixed-citation publication-type="journal"><name><surname>Wilczynski</surname><given-names>NL</given-names></name><name><surname>Haynes</surname><given-names>RB</given-names></name><article-title>Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey</article-title><source>BMC Med</source><year>2004</year><volume>2</volume><fpage>23</fpage><pub-id pub-id-type="pmid">15189561</pub-id></mixed-citation></ref><ref id="B32"><mixed-citation publication-type="journal"><name><surname>Montori</surname><given-names>VM</given-names></name><name><surname>Wilczynski</surname><given-names>NL</given-names></name><name><surname>Morgan</surname><given-names>D</given-names></name><name><surname>Haynes</surname><given-names>RB</given-names></name><article-title>Optimal search strategies for retrieving systematic reviews from Medline: analytical survey</article-title><source>BMJ</source><year>2005</year><volume>330</volume><fpage>68</fpage><pub-id pub-id-type="pmid">15619601</pub-id></mixed-citation></ref><ref id="B33"><mixed-citation publication-type="journal"><name><surname>Wilczynski</surname><given-names>NL</given-names></name><name><surname>Haynes</surname><given-names>RB</given-names></name><name><surname>Lavis</surname><given-names>JN</given-names></name><name><surname>Ramkissoonsingh</surname><given-names>R</given-names></name><name><surname>Arnold-Oatley</surname><given-names>AE</given-names></name><article-title>Optimal search strategies for detecting health services research studies in MEDLINE</article-title><source>CMAJ</source><year>2004</year><volume>171</volume><fpage>1179</fpage><lpage>1185</lpage><pub-id pub-id-type="pmid">15534310</pub-id></mixed-citation></ref><ref id="B34"><mixed-citation publication-type="journal"><name><surname>Haynes</surname><given-names>RB</given-names></name><name><surname>Wilczynski</surname><given-names>NL</given-names></name><article-title>Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey</article-title><source>BMJ</source><year>2004</year><volume>328</volume><fpage>1040</fpage><pub-id pub-id-type="pmid">15073027</pub-id></mixed-citation></ref><ref id="B35"><mixed-citation publication-type="journal"><name><surname>Shojania</surname><given-names>KG</given-names></name><name><surname>Bero</surname><given-names>LA</given-names></name><article-title>Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy</article-title><source>Eff Clin Pract</source><year>2000</year><volume>4</volume><fpage>157</fpage><lpage>162</lpage><pub-id pub-id-type="pmid">11525102</pub-id></mixed-citation></ref><ref id="B36"><mixed-citation publication-type="journal"><name><surname>Wilczynski</surname><given-names>NL</given-names></name><name><surname>McKibbon</surname><given-names>KA</given-names></name><name><surname>Haynes</surname><given-names>RB</given-names></name><article-title>Sensitive Clinical Queries retrieved relevant systematic reviews as well as primary studies: an analytic survey</article-title><source>J Clin Epidemiol</source><year>2011</year><volume>64</volume><fpage>1341</fpage><lpage>1349</lpage><pub-id pub-id-type="pmid">21775104</pub-id></mixed-citation></ref><ref id="B37"><mixed-citation publication-type="journal"><name><surname>Simmons</surname><given-names>RF</given-names></name><article-title>Answering English questions by computer: a survey</article-title><source>Commun ACM</source><year>1965</year><volume>8</volume><fpage>53</fpage><lpage>70</lpage></mixed-citation></ref><ref id="B38"><mixed-citation publication-type="journal"><name><surname>Kwok</surname><given-names>C</given-names></name><name><surname>Etzioni</surname><given-names>O</given-names></name><name><surname>Weld</surname><given-names>DS</given-names></name><article-title>Scaling question answering to the web</article-title><source>ACM Trans Inform Syst</source><year>2001</year><volume>19</volume><fpage>242</fpage><lpage>262</lpage></mixed-citation></ref><ref id="B39"><mixed-citation publication-type="book"><name><surname>Dumais</surname><given-names>S</given-names></name><name><surname>Banko</surname><given-names>M</given-names></name><name><surname>Brill</surname><given-names>E</given-names></name><name><surname>Lin</surname><given-names>J</given-names></name><name><surname>Ng</surname><given-names>A</given-names></name><article-title>Web question answering: Is more always better?</article-title><source>Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</source><year>2002</year><publisher-name>Tampere, Finland. New York: ACM</publisher-name><fpage>291</fpage><lpage>298</lpage></mixed-citation></ref><ref id="B40"><mixed-citation publication-type="book"><name><surname>Collaboration</surname><given-names>RTC</given-names></name><source>Review Manager (RevMan). 4.2 for Windows</source><year>2003</year><publisher-name>Oxford, England: The Cochrane Collaboration</publisher-name></mixed-citation></ref><ref id="B41"><mixed-citation publication-type="journal"><name><surname>Higgins</surname><given-names>J</given-names></name><name><surname>Churchill</surname><given-names>R</given-names></name><name><surname>Tovey</surname><given-names>D</given-names></name><name><surname>Lasserson</surname><given-names>T</given-names></name><name><surname>Chandler</surname><given-names>J</given-names></name><article-title>Update on the MECIR project: methodological expectations for Cochrane intervention</article-title><source>Cochrane Meth</source><year>2011</year><volume>suppl. 1</volume><fpage>2</fpage></mixed-citation></ref><ref id="B42"><mixed-citation publication-type="journal"><name><surname>Tsafnat</surname><given-names>G</given-names></name><name><surname>Coiera</surname><given-names>E</given-names></name><article-title>Computational reasoning across multiple models</article-title><source>J Am Med Info Assoc</source><year>2009</year><volume>16</volume><fpage>768</fpage></mixed-citation></ref><ref id="B43"><mixed-citation publication-type="journal"><name><surname>Sim</surname><given-names>I</given-names></name><name><surname>Detmer</surname><given-names>DE</given-names></name><article-title>Beyond trial registration: a global trial bank for clinical trial reporting</article-title><source>PLoS Med</source><year>2005</year><volume>2</volume><fpage>e365</fpage><pub-id pub-id-type="pmid">16220999</pub-id></mixed-citation></ref><ref id="B44"><mixed-citation publication-type="other"><name><surname>Sim</surname><given-names>I</given-names></name><name><surname>Tu</surname><given-names>SW</given-names></name><name><surname>Carini</surname><given-names>S</given-names></name><name><surname>Lehmann</surname><given-names>HP</given-names></name><name><surname>Pollock</surname><given-names>BH</given-names></name><name><surname>Peleg</surname><given-names>M</given-names></name><name><surname>Wittkowski</surname><given-names>KM</given-names></name><article-title>The Ontology of Clinical Research (OCRe): an informatics foundation for the science of clinical research</article-title><source>J Biomed Inform</source><year>2013</year></mixed-citation></ref><ref id="B45"><mixed-citation publication-type="journal"><name><surname>Balk</surname><given-names>EM</given-names></name><name><surname>Chung</surname><given-names>M</given-names></name><name><surname>Chen</surname><given-names>ML</given-names></name><name><surname>Chang</surname><given-names>LKW</given-names></name><name><surname>Trikalinos</surname><given-names>TA</given-names></name><article-title>Data extraction from machine-translated versus original language randomized trial reports: a comparative study</article-title><source>Syst Rev</source><year>2013</year><volume>2</volume><fpage>97</fpage><pub-id pub-id-type="pmid">24199894</pub-id></mixed-citation></ref><ref id="B46"><mixed-citation publication-type="journal"><name><surname>Greenhalgh</surname><given-names>T</given-names></name><name><surname>Peacock</surname><given-names>R</given-names></name><article-title>Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources</article-title><source>BMJ</source><year>2005</year><volume>331</volume><fpage>1064</fpage><pub-id pub-id-type="pmid">16230312</pub-id></mixed-citation></ref><ref id="B47"><mixed-citation publication-type="journal"><name><surname>Kuper</surname><given-names>H</given-names></name><name><surname>Nicholson</surname><given-names>A</given-names></name><name><surname>Hemingway</surname><given-names>H</given-names></name><article-title>Searching for observational studies: what does citation tracking add to PubMed?</article-title><source>A case study in depression and coronary heart disease. BMC Med Res Methodol</source><year>2006</year><volume>6</volume><fpage>4</fpage></mixed-citation></ref><ref id="B48"><mixed-citation publication-type="journal"><name><surname>Cepeda</surname><given-names>MS</given-names></name><name><surname>Lobanov</surname><given-names>V</given-names></name><name><surname>Berlin</surname><given-names>JA</given-names></name><article-title>Using Sherlock and ClinicalTrials.gov data to understand nocebo effects and adverse event dropout rates in the placebo arm</article-title><source>J Pain</source><year>2013</year><volume>14</volume><fpage>999</fpage><pub-id pub-id-type="pmid">23993418</pub-id></mixed-citation></ref><ref id="B49"><mixed-citation publication-type="journal"><name><surname>Lowe</surname><given-names>HJ</given-names></name><name><surname>Barnett</surname><given-names>GO</given-names></name><article-title>Understanding and using the medical subject headings (MeSH) vocabulary to perform literature searches</article-title><source>JAMA</source><year>1994</year><volume>271</volume><fpage>1103</fpage><lpage>1108</lpage><pub-id pub-id-type="pmid">8151853</pub-id></mixed-citation></ref><ref id="B50"><mixed-citation publication-type="journal"><name><surname>Robinson</surname><given-names>KA</given-names></name><name><surname>Dickersin</surname><given-names>K</given-names></name><article-title>Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed</article-title><source>Int J Epidemiol</source><year>2002</year><volume>31</volume><fpage>150</fpage><lpage>153</lpage><pub-id pub-id-type="pmid">11914311</pub-id></mixed-citation></ref><ref id="B51"><mixed-citation publication-type="journal"><name><surname>Doshi</surname><given-names>P</given-names></name><name><surname>Jones</surname><given-names>M</given-names></name><name><surname>Jefferson</surname><given-names>T</given-names></name><article-title>Rethinking credible evidence synthesis</article-title><source>BMJ</source><year>2012</year><volume>344</volume><fpage>d7898</fpage><pub-id pub-id-type="pmid">22252039</pub-id></mixed-citation></ref><ref id="B52"><mixed-citation publication-type="journal"><name><surname>Eysenbach</surname><given-names>G</given-names></name><name><surname>Tuische</surname><given-names>J</given-names></name><name><surname>Diepgen</surname><given-names>TL</given-names></name><article-title>Evaluation of the usefulness of Internet searches to identify unpublished clinical trials for systematic reviews</article-title><source>Med Inform Internet Med</source><year>2001</year><volume>26</volume><fpage>203</fpage><lpage>218</lpage><pub-id pub-id-type="pmid">11706930</pub-id></mixed-citation></ref><ref id="B53"><mixed-citation publication-type="journal"><name><surname>McKibbon</surname><given-names>KA</given-names></name><article-title>Systematic reviews and librarians</article-title><source>Libr Trends</source><year>2006</year><volume>55</volume><fpage>202</fpage><lpage>215</lpage></mixed-citation></ref><ref id="B54"><mixed-citation publication-type="journal"><name><surname>Harris</surname><given-names>MR</given-names></name><article-title>The librarian's roles in the systematic review process: a case study</article-title><source>J Med Libr Assoc</source><year>2005</year><volume>93</volume><fpage>81</fpage><lpage>87</lpage><pub-id pub-id-type="pmid">15685279</pub-id></mixed-citation></ref><ref id="B55"><mixed-citation publication-type="journal"><name><surname>Beller</surname><given-names>EM</given-names></name><name><surname>Chen</surname><given-names>JK</given-names></name><name><surname>Wang</surname><given-names>UL</given-names></name><name><surname>Glasziou</surname><given-names>PP</given-names></name><article-title>Are systematic reviews up-to-date at the time of publication?</article-title><source>Syst Rev</source><year>2013</year><volume>2</volume><fpage>36</fpage><pub-id pub-id-type="pmid">23714302</pub-id></mixed-citation></ref><ref id="B56"><mixed-citation publication-type="journal"><name><surname>Hull</surname><given-names>D</given-names></name><name><surname>Pettifer</surname><given-names>SR</given-names></name><name><surname>Kell</surname><given-names>DB</given-names></name><article-title>Defrosting the digital library: bibliographic tools for the next generation web</article-title><source>PLoS Comput Biol</source><year>2008</year><volume>4</volume><fpage>e1000204</fpage><pub-id pub-id-type="pmid">18974831</pub-id></mixed-citation></ref><ref id="B57"><mixed-citation publication-type="journal"><name><surname>Chalmers</surname><given-names>I</given-names></name><name><surname>Glasziou</surname><given-names>P</given-names></name><name><surname>Godlee</surname><given-names>F</given-names></name><article-title>All trials must be registered and the results published</article-title><source>BMJ</source><year>2013</year><volume>346</volume><fpage>105</fpage></mixed-citation></ref><ref id="B58"><mixed-citation publication-type="journal"><name><surname>Huser</surname><given-names>V</given-names></name><name><surname>Cimino</surname><given-names>JJ</given-names></name><article-title>Evaluating adherence to the International Committee of Medical Journal Editors' policy of mandatory, timely clinical trial registration</article-title><source>J Am Med Inform Assoc</source><year>2013</year><volume>20</volume><fpage>e169</fpage><lpage>e174</lpage><pub-id pub-id-type="pmid">23396544</pub-id></mixed-citation></ref><ref id="B59"><mixed-citation publication-type="other"><collab>World Health Organization</collab><source>International clinical trials registry platform (ICTRP)</source><year>2008</year><comment>Welcome to the WHO International Clinical Trials Registry Platform Obtenido de: <ext-link ext-link-type="uri" xlink:href="http://www.who.int/ictrp/en/">http://www.who.int/ictrp/en/</ext-link> (Consulta: 31 may 2006)</comment></mixed-citation></ref><ref id="B60"><mixed-citation publication-type="journal"><name><surname>Wilkins</surname><given-names>T</given-names></name><name><surname>Gillies</surname><given-names>RA</given-names></name><name><surname>Davies</surname><given-names>K</given-names></name><article-title>EMBASE versus MEDLINE for family medicine searches: can MEDLINE searches find the forest or a tree?</article-title><source>Can Fam Physician</source><year>2005</year><volume>51</volume><fpage>848</fpage><lpage>849</lpage><pub-id pub-id-type="pmid">16926954</pub-id></mixed-citation></ref><ref id="B61"><mixed-citation publication-type="book"><name><surname>Wolf</surname><given-names>F</given-names></name><name><surname>Grum</surname><given-names>C</given-names></name><name><surname>Bara</surname><given-names>A</given-names></name><name><surname>Milan</surname><given-names>S</given-names></name><name><surname>Jones</surname><given-names>P</given-names></name><article-title>Comparison of Medline and Embase retrieval of RCTs of the effects of educational interventions on asthma-related outcomes</article-title><source>Cochrane Colloquium Abstracts Journal</source><year>1995</year><publisher-name>Oslo, Norway: Wiley Blackwell</publisher-name></mixed-citation></ref><ref id="B62"><mixed-citation publication-type="journal"><name><surname>Woods</surname><given-names>D</given-names></name><name><surname>Trewheellar</surname><given-names>K</given-names></name><article-title>Medline and Embase complement each other in literature searches</article-title><source>BMJ</source><year>1998</year><volume>316</volume><fpage>1166</fpage><pub-id pub-id-type="pmid">9552968</pub-id></mixed-citation></ref><ref id="B63"><mixed-citation publication-type="journal"><name><surname>Shariff</surname><given-names>SZ</given-names></name><name><surname>Bejaimal</surname><given-names>SA</given-names></name><name><surname>Sontrop</surname><given-names>JM</given-names></name><name><surname>Iansavichus</surname><given-names>AV</given-names></name><name><surname>Haynes</surname><given-names>RB</given-names></name><name><surname>Weir</surname><given-names>MA</given-names></name><name><surname>Garg</surname><given-names>AX</given-names></name><article-title>Retrieving clinical evidence: a comparison of PubMed and Google Scholar for Quick Clinical Searches</article-title><source>J Med Internet Res</source><year>2013</year><volume>15</volume><fpage>e164</fpage><pub-id pub-id-type="pmid">23948488</pub-id></mixed-citation></ref><ref id="B64"><mixed-citation publication-type="journal"><name><surname>Carpineto</surname><given-names>C</given-names></name><name><surname>Romano</surname><given-names>G</given-names></name><article-title>A survey of automatic query expansion in information retrieval</article-title><source>ACM Comput Survey</source><year>2012</year><volume>44</volume><fpage>1</fpage></mixed-citation></ref><ref id="B65"><mixed-citation publication-type="journal"><name><surname>Chatterjee</surname><given-names>R</given-names></name><article-title>An analytical assessment on document clustering</article-title><source>Int J Comp Net Inform Sec</source><year>2012</year><volume>4</volume><fpage>63</fpage></mixed-citation></ref><ref id="B66"><mixed-citation publication-type="journal"><name><surname>Boyack</surname><given-names>KW</given-names></name><name><surname>Newman</surname><given-names>D</given-names></name><name><surname>Duhon</surname><given-names>RJ</given-names></name><name><surname>Klavans</surname><given-names>R</given-names></name><name><surname>Patek</surname><given-names>M</given-names></name><name><surname>Biberstine</surname><given-names>JR</given-names></name><name><surname>Schijvenaars</surname><given-names>B</given-names></name><name><surname>Skupin</surname><given-names>A</given-names></name><name><surname>Ma</surname><given-names>N</given-names></name><name><surname>Börner</surname><given-names>K</given-names></name><article-title>Clustering more than two million biomedical publications: Comparing the accuracies of nine text-based similarity approaches</article-title><source>PLoS One</source><year>2011</year><volume>6</volume><fpage>e18029</fpage><pub-id pub-id-type="pmid">21437291</pub-id></mixed-citation></ref><ref id="B67"><mixed-citation publication-type="journal"><name><surname>Aljaber</surname><given-names>B</given-names></name><name><surname>Stokes</surname><given-names>N</given-names></name><name><surname>Bailey</surname><given-names>J</given-names></name><name><surname>Pei</surname><given-names>J</given-names></name><article-title>Document clustering of scientific texts using citation contexts</article-title><source>Inform Retrieval</source><year>2010</year><volume>13</volume><fpage>101</fpage><lpage>131</lpage></mixed-citation></ref><ref id="B68"><mixed-citation publication-type="journal"><name><surname>Smalheiser</surname><given-names>NR</given-names></name><name><surname>Lin</surname><given-names>C</given-names></name><name><surname>Jia</surname><given-names>L</given-names></name><name><surname>Jiang</surname><given-names>Y</given-names></name><name><surname>Cohen</surname><given-names>AM</given-names></name><name><surname>Yu</surname><given-names>C</given-names></name><name><surname>Davis</surname><given-names>JM</given-names></name><name><surname>Adams</surname><given-names>CE</given-names></name><name><surname>McDonagh</surname><given-names>MS</given-names></name><name><surname>Meng</surname><given-names>W</given-names></name><article-title>Design and implementation of Metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers</article-title><source>Health Inform Sci Syst</source><year>2014</year><volume>2</volume><fpage>1</fpage></mixed-citation></ref><ref id="B69"><mixed-citation publication-type="journal"><name><surname>Bracke</surname><given-names>PJ</given-names></name><name><surname>Howse</surname><given-names>DK</given-names></name><name><surname>Keim</surname><given-names>SM</given-names></name><article-title>Evidence-based Medicine Search: a customizable federated search engine</article-title><source>J Med Libr Assoc</source><year>2008</year><volume>96</volume><fpage>108</fpage><pub-id pub-id-type="pmid">18379665</pub-id></mixed-citation></ref><ref id="B70"><mixed-citation publication-type="journal"><name><surname>Coiera</surname><given-names>E</given-names></name><name><surname>Walther</surname><given-names>M</given-names></name><name><surname>Nguyen</surname><given-names>K</given-names></name><name><surname>Lovell</surname><given-names>NH</given-names></name><article-title>Architecture for knowledge-based and federated search of online clinical evidence</article-title><source>J Med Internet Res</source><year>2005</year><volume>7</volume><fpage>e52</fpage><pub-id pub-id-type="pmid">16403716</pub-id></mixed-citation></ref><ref id="B71"><mixed-citation publication-type="journal"><name><surname>Magrabi</surname><given-names>F</given-names></name><name><surname>Westbrook</surname><given-names>JI</given-names></name><name><surname>Kidd</surname><given-names>MR</given-names></name><name><surname>Day</surname><given-names>RO</given-names></name><name><surname>Coiera</surname><given-names>E</given-names></name><article-title>Long-term patterns of online evidence retrieval use in general practice: a 12-month study</article-title><source>J Med Internet Res</source><year>2008</year><volume>10</volume><fpage>e6</fpage><pub-id pub-id-type="pmid">18353750</pub-id></mixed-citation></ref><ref id="B72"><mixed-citation publication-type="journal"><name><surname>Meats</surname><given-names>E</given-names></name><name><surname>Brassey</surname><given-names>J</given-names></name><name><surname>Heneghan</surname><given-names>C</given-names></name><name><surname>Glasziou</surname><given-names>P</given-names></name><article-title>Using the Turning Research Into Practice (TRIP) database: how do clinicians really search?</article-title><source>J Med Libr Assoc</source><year>2007</year><volume>95</volume><fpage>156</fpage><lpage>163</lpage><pub-id pub-id-type="pmid">17443248</pub-id></mixed-citation></ref><ref id="B73"><mixed-citation publication-type="journal"><name><surname>Whetzel</surname><given-names>PL</given-names></name><name><surname>Noy</surname><given-names>NF</given-names></name><name><surname>Shah</surname><given-names>NH</given-names></name><name><surname>Alexander</surname><given-names>PR</given-names></name><name><surname>Nyulas</surname><given-names>C</given-names></name><name><surname>Tudorache</surname><given-names>T</given-names></name><name><surname>Musen</surname><given-names>MA</given-names></name><article-title>BioPortal: enhanced functionality via new Web services from the National Center for Biomedical Ontology to access and use ontologies in software applications</article-title><source>Nucleic Acids Res</source><year>2011</year><volume>39</volume><fpage>W541</fpage><lpage>W545</lpage><pub-id pub-id-type="pmid">21672956</pub-id></mixed-citation></ref><ref id="B74"><mixed-citation publication-type="journal"><name><surname>De Angelis</surname><given-names>C</given-names></name><name><surname>Drazen</surname><given-names>JM</given-names></name><name><surname>Frizelle</surname><given-names>FA</given-names></name><name><surname>Haug</surname><given-names>C</given-names></name><name><surname>Hoey</surname><given-names>J</given-names></name><name><surname>Horton</surname><given-names>R</given-names></name><name><surname>Kotzin</surname><given-names>S</given-names></name><name><surname>Laine</surname><given-names>C</given-names></name><name><surname>Marusic</surname><given-names>A</given-names></name><name><surname>Overbeke</surname><given-names>AJP</given-names></name><article-title>Clinical trial registration: a statement from the International Committee of Medical Journal Editors</article-title><source>N Engl J Med</source><year>2004</year><volume>351</volume><fpage>1250</fpage><lpage>1251</lpage><pub-id pub-id-type="pmid">15356289</pub-id></mixed-citation></ref><ref id="B75"><mixed-citation publication-type="other"><collab>Health UNIo</collab><source>ClinicalTrials. gov</source><year>2009</year><comment><ext-link ext-link-type="uri" xlink:href="http://www.clinicaltrials.gov">http://www.clinicaltrials.gov</ext-link></comment></mixed-citation></ref><ref id="B76"><mixed-citation publication-type="other"><collab>European Union Clinical Trials Register</collab><comment><ext-link ext-link-type="uri" xlink:href="https://www.clinicaltrialsregister.eu/">https://www.clinicaltrialsregister.eu/</ext-link></comment></mixed-citation></ref><ref id="B77"><mixed-citation publication-type="journal"><name><surname>Askie</surname><given-names>LM</given-names></name><article-title>Australian New Zealand Clinical Trials Registry: history and growth</article-title><source>J Evid-Based Med</source><year>2011</year><volume>4</volume><fpage>185</fpage><lpage>187</lpage><pub-id pub-id-type="pmid">23672724</pub-id></mixed-citation></ref><ref id="B78"><mixed-citation publication-type="journal"><name><surname>Qi</surname><given-names>X-S</given-names></name><name><surname>Bai</surname><given-names>M</given-names></name><name><surname>Yang</surname><given-names>Z-P</given-names></name><name><surname>Ren</surname><given-names>W-R</given-names></name><name><surname>McFarland</surname><given-names>LV</given-names></name><name><surname>Goh</surname><given-names>S</given-names></name><name><surname>Linz</surname><given-names>K</given-names></name><name><surname>Miller</surname><given-names>BJ</given-names></name><name><surname>Zhang</surname><given-names>H</given-names></name><name><surname>Gao</surname><given-names>C</given-names></name><article-title>Duplicates in systematic reviews: A critical, but often neglected issue</article-title><source>World J of Meta-Anal</source><year>2013</year><volume>3</volume><fpage>97</fpage><lpage>101</lpage></mixed-citation></ref><ref id="B79"><mixed-citation publication-type="journal"><name><surname>Qi</surname><given-names>X</given-names></name><name><surname>Yang</surname><given-names>M</given-names></name><name><surname>Ren</surname><given-names>W</given-names></name><name><surname>Jia</surname><given-names>J</given-names></name><name><surname>Wang</surname><given-names>J</given-names></name><name><surname>Han</surname><given-names>G</given-names></name><name><surname>Fan</surname><given-names>D</given-names></name><article-title>Find duplicates among the PubMed, EMBASE, and Cochrane Library databases in systematic review</article-title><source>PLoS One</source><year>2013</year><volume>8</volume><fpage>e71838</fpage><pub-id pub-id-type="pmid">23977157</pub-id></mixed-citation></ref><ref id="B80"><mixed-citation publication-type="journal"><name><surname>Elmagarmid</surname><given-names>AK</given-names></name><name><surname>Ipeirotis</surname><given-names>PG</given-names></name><name><surname>Verykios</surname><given-names>VS</given-names></name><article-title>Duplicate record detection: a survey</article-title><source>IEEE Trans Knowl Data Eng</source><year>2007</year><volume>19</volume><fpage>1</fpage><lpage>16</lpage></mixed-citation></ref><ref id="B81"><mixed-citation publication-type="journal"><name><surname>Aabenhus</surname><given-names>R</given-names></name><name><surname>Jensen</surname><given-names>JU</given-names></name><name><surname>Cals</surname><given-names>JW</given-names></name><article-title>Incorrect inclusion of individual studies and methodological flaws in systematic review and meta-analysis</article-title><source>Br J Gen Pract</source><year>2014</year><volume>64</volume><fpage>221</fpage><lpage>222</lpage><pub-id pub-id-type="pmid">24771816</pub-id></mixed-citation></ref><ref id="B82"><mixed-citation publication-type="book"><collab>REUTERS T</collab><source>EndNote®</source><year>2011</year><publisher-name>Bibliographies Made Easy™</publisher-name><comment><ext-link ext-link-type="uri" xlink:href="http://www.endnote.com">http://www.endnote.com</ext-link></comment></mixed-citation></ref><ref id="B83"><mixed-citation publication-type="book"><name><surname>ResearchSoft</surname><given-names>I</given-names></name><source>ProCite Version 5</source><year>1999</year><publisher-name>Berkeley, CA: Institute for Scientific Information</publisher-name></mixed-citation></ref><ref id="B84"><mixed-citation publication-type="journal"><name><surname>Jiang</surname><given-names>Y</given-names></name><name><surname>Lin</surname><given-names>C</given-names></name><name><surname>Meng</surname><given-names>W</given-names></name><name><surname>Yu</surname><given-names>C</given-names></name><name><surname>Cohen</surname><given-names>AM</given-names></name><name><surname>Smalheiser</surname><given-names>NR</given-names></name><article-title>Rule-based deduplication of article records from bibliographic databases</article-title><source>Database</source><year>2014</year><volume>bat086</volume></mixed-citation></ref><ref id="B85"><mixed-citation publication-type="book"><name><surname>Councill</surname><given-names>IG</given-names></name><name><surname>Giles</surname><given-names>CL</given-names></name><name><surname>Kan</surname><given-names>M-Y</given-names></name><article-title>ParsCit: an open-source CRF reference string parsing package</article-title><source>Language Resources and Evaluation Conference</source><year>2008</year><publisher-name>Morocco: Marrakech</publisher-name><fpage>28</fpage><lpage>30</lpage></mixed-citation></ref><ref id="B86"><mixed-citation publication-type="book"><name><surname>Granitzer</surname><given-names>M</given-names></name><name><surname>Hristakeva</surname><given-names>M</given-names></name><name><surname>Jack</surname><given-names>K</given-names></name><name><surname>Knight</surname><given-names>R</given-names></name><article-title>A comparison of metadata extraction techniques for crowdsourced bibliographic metadata management</article-title><source>Proceedings of the 27th Annual ACM Symposium on Applied Computing</source><year>2012</year><publisher-name>Riva (Trento), Italy. New York: ACM</publisher-name><fpage>962</fpage><lpage>964</lpage></mixed-citation></ref><ref id="B87"><mixed-citation publication-type="book"><name><surname>Nhat</surname><given-names>HDH</given-names></name><name><surname>Bysani</surname><given-names>P</given-names></name><article-title>Linking citations to their bibliographic references</article-title><source>Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries</source><year>2012</year><publisher-name>Stroudsburg: PA Association for Computational Linguistics</publisher-name><fpage>110</fpage><lpage>113</lpage></mixed-citation></ref><ref id="B88"><mixed-citation publication-type="book"><name><surname>Saleem</surname><given-names>O</given-names></name><name><surname>Latif</surname><given-names>S</given-names></name><article-title>Information extraction from research papers by data integration and data validation from multiple header extraction sources</article-title><source>Proceedings of the World Congress on Engineering and Computer Science</source><year>2012</year><publisher-name>San Francisco, USA</publisher-name></mixed-citation></ref><ref id="B89"><mixed-citation publication-type="book"><name><surname>Day</surname><given-names>M-Y</given-names></name><name><surname>Tsai</surname><given-names>T-H</given-names></name><name><surname>Sung</surname><given-names>C-L</given-names></name><name><surname>Lee</surname><given-names>C-W</given-names></name><name><surname>Wu</surname><given-names>S-H</given-names></name><name><surname>Ong</surname><given-names>C-S</given-names></name><name><surname>Hsu</surname><given-names>W-L</given-names></name><article-title>A knowledge-based approach to citation extraction</article-title><source>IEEE International Conference on Information Reuse and Integration Conference</source><year>2005</year><publisher-name>Las Vegas, Nevada, USA</publisher-name><fpage>50</fpage><lpage>55</lpage></mixed-citation></ref><ref id="B90"><mixed-citation publication-type="journal"><name><surname>Errami</surname><given-names>M</given-names></name><name><surname>Hicks</surname><given-names>JM</given-names></name><name><surname>Fisher</surname><given-names>W</given-names></name><name><surname>Trusty</surname><given-names>D</given-names></name><name><surname>Wren</surname><given-names>JD</given-names></name><name><surname>Long</surname><given-names>TC</given-names></name><name><surname>Garner</surname><given-names>HR</given-names></name><article-title>Déjà vu—a study of duplicate citations in Medline</article-title><source>Bioinformatics</source><year>2008</year><volume>24</volume><fpage>243</fpage><lpage>249</lpage><pub-id pub-id-type="pmid">18056062</pub-id></mixed-citation></ref><ref id="B91"><mixed-citation publication-type="journal"><name><surname>Zhang</surname><given-names>HY</given-names></name><article-title>CrossCheck: an effective tool for detecting plagiarism</article-title><source>Learned Publishing</source><year>2010</year><volume>23</volume><fpage>9</fpage><lpage>14</lpage></mixed-citation></ref><ref id="B92"><mixed-citation publication-type="book"><name><surname>Meuschke</surname><given-names>N</given-names></name><name><surname>Gipp</surname><given-names>B</given-names></name><name><surname>Breitinger</surname><given-names>C</given-names></name><name><surname>Berkeley</surname><given-names>U</given-names></name><article-title>CitePlag: a citation-based plagiarism detection system prototype</article-title><source>Proceedings of the 5th International Plagiarism Conference</source><year>2012</year><publisher-name>Newcastle, UK</publisher-name></mixed-citation></ref><ref id="B93"><mixed-citation publication-type="book"><name><surname>Chung</surname><given-names>GY</given-names></name><name><surname>Coiera</surname><given-names>E</given-names></name><article-title>A study of structured clinical abstracts and the semantic classification of sentences</article-title><source>Proceedings of the Workshop on BioNLP 2007: Biological, Translational, and Clinical Language Processing</source><year>2007</year><publisher-name>Stroudsburg, PA, USA: Association for Computational Linguistics</publisher-name><fpage>121</fpage><lpage>128</lpage></mixed-citation></ref><ref id="B94"><mixed-citation publication-type="journal"><name><surname>Thomas</surname><given-names>J</given-names></name><name><surname>McNaught</surname><given-names>J</given-names></name><name><surname>Ananiadou</surname><given-names>S</given-names></name><article-title>Applications of text mining within systematic reviews</article-title><source>Res Synth Meth</source><year>2011</year><volume>2</volume><fpage>1</fpage><lpage>14</lpage></mixed-citation></ref><ref id="B95"><mixed-citation publication-type="journal"><name><surname>Kiritchenko</surname><given-names>S</given-names></name><name><surname>de Bruijn</surname><given-names>B</given-names></name><name><surname>Carini</surname><given-names>S</given-names></name><name><surname>Martin</surname><given-names>J</given-names></name><name><surname>Sim</surname><given-names>I</given-names></name><article-title>ExaCT: automatic extraction of clinical trial characteristics from journal publications</article-title><source>BMC Med Inform Decis Mak</source><year>2010</year><volume>10</volume><fpage>56</fpage><pub-id pub-id-type="pmid">20920176</pub-id></mixed-citation></ref><ref id="B96"><mixed-citation publication-type="journal"><name><surname>Kim</surname><given-names>SN</given-names></name><name><surname>Martinez</surname><given-names>D</given-names></name><name><surname>Cavedon</surname><given-names>L</given-names></name><name><surname>Yencken</surname><given-names>L</given-names></name><article-title>Automatic classification of sentences to support Evidence Based Medicine</article-title><source>BMC Bioinform</source><year>2011</year><volume>12</volume><issue>Suppl 2</issue><fpage>S5</fpage></mixed-citation></ref><ref id="B97"><mixed-citation publication-type="journal"><name><surname>Zhao</surname><given-names>J</given-names></name><name><surname>Bysani</surname><given-names>P</given-names></name><name><surname>Kan</surname><given-names>MY</given-names></name><article-title>Exploiting classification correlations for the extraction of evidence-based practice information</article-title><source>AMIA Annu Symp Proc</source><year>2012</year><volume>2012</volume><fpage>1070</fpage><lpage>1078</lpage><pub-id pub-id-type="pmid">23304383</pub-id></mixed-citation></ref><ref id="B98"><mixed-citation publication-type="journal"><name><surname>Hansen</surname><given-names>MJ</given-names></name><name><surname>Rasmussen</surname><given-names>NØ</given-names></name><name><surname>Chung</surname><given-names>G</given-names></name><article-title>A method of extracting the number of trial participants from abstracts describing randomized controlled trials</article-title><source>J Telemed Telecare</source><year>2008</year><volume>14</volume><fpage>354</fpage><lpage>358</lpage><pub-id pub-id-type="pmid">18852316</pub-id></mixed-citation></ref><ref id="B99"><mixed-citation publication-type="book"><name><surname>Thomas</surname><given-names>J</given-names></name><name><surname>Brunton</surname><given-names>J</given-names></name><name><surname>Graziosi</surname><given-names>S</given-names></name><source>EPPI-Reviewer 4.0: software for research synthesis</source><year>2010</year><publisher-name>EPPI-Centre, Social Science Research Unit, Institute of Education, University of London UK</publisher-name></mixed-citation></ref><ref id="B100"><mixed-citation publication-type="other"><collab>DistillerSR</collab><comment><ext-link ext-link-type="uri" xlink:href="http://www.systematic-review.net">http://www.systematic-review.net</ext-link></comment></mixed-citation></ref><ref id="B101"><mixed-citation publication-type="other"><collab>Covidence</collab><comment><ext-link ext-link-type="uri" xlink:href="http://www.covidence.org">http://www.covidence.org</ext-link></comment></mixed-citation></ref><ref id="B102"><mixed-citation publication-type="other"><name><surname>García Adeva</surname><given-names>JJ</given-names></name><name><surname>Pikatza Atxa</surname><given-names>JM</given-names></name><name><surname>Ubeda Carrillo</surname><given-names>M</given-names></name><name><surname>Ansuategi Zengotitabengoa</surname><given-names>E</given-names></name><article-title>Automatic text classification to support systematic reviews in medicine</article-title><source>Expert Syst Appl</source><year>2013</year><comment>in press</comment></mixed-citation></ref><ref id="B103"><mixed-citation publication-type="journal"><name><surname>Wallace</surname><given-names>BC</given-names></name><name><surname>Trikalinos</surname><given-names>TA</given-names></name><name><surname>Lau</surname><given-names>J</given-names></name><name><surname>Brodley</surname><given-names>C</given-names></name><name><surname>Schmid</surname><given-names>CH</given-names></name><article-title>Semi-automated screening of biomedical citations for systematic reviews</article-title><source>BMC Bioinform</source><year>2010</year><volume>11</volume><fpage>55</fpage></mixed-citation></ref><ref id="B104"><mixed-citation publication-type="book"><name><surname>Wallace</surname><given-names>BC</given-names></name><name><surname>Small</surname><given-names>K</given-names></name><name><surname>Brodley</surname><given-names>CE</given-names></name><name><surname>Lau</surname><given-names>J</given-names></name><name><surname>Trikalinos</surname><given-names>TA</given-names></name><article-title>Deploying an interactive machine learning system in an evidence-based practice center: abstrackr</article-title><source>Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium</source><year>2012</year><publisher-name>Miamy, Florida, USA. New York: ACM</publisher-name><fpage>819</fpage><lpage>824</lpage></mixed-citation></ref><ref id="B105"><mixed-citation publication-type="journal"><name><surname>Ananiadou</surname><given-names>S</given-names></name><name><surname>Rea</surname><given-names>B</given-names></name><name><surname>Okazaki</surname><given-names>N</given-names></name><name><surname>Procter</surname><given-names>R</given-names></name><name><surname>Thomas</surname><given-names>J</given-names></name><article-title>Supporting systematic reviews using text mining</article-title><source>Soc Sci Comp Rev</source><year>2009</year><volume>27</volume><fpage>509</fpage><lpage>523</lpage></mixed-citation></ref><ref id="B106"><mixed-citation publication-type="journal"><name><surname>Cohen</surname><given-names>AM</given-names></name><name><surname>Ambert</surname><given-names>K</given-names></name><name><surname>McDonagh</surname><given-names>M</given-names></name><article-title>Studying the potential impact of automated document classification on scheduling a systematic review update</article-title><source>BMC Med Inform Decis Mak</source><year>2012</year><volume>12</volume><fpage>33</fpage><pub-id pub-id-type="pmid">22515596</pub-id></mixed-citation></ref><ref id="B107"><mixed-citation publication-type="other"><name><surname>Sarker</surname><given-names>A</given-names></name><name><surname>Mollá-Aliod</surname><given-names>D</given-names></name><article-title>A rule based approach for automatic identification of publication types of medical papers</article-title><source>Proceedings of the ADCS Annual Symposium</source><year>2010</year><fpage>84</fpage><lpage>88</lpage></mixed-citation></ref><ref id="B108"><mixed-citation publication-type="book"><name><surname>Kim</surname><given-names>MH</given-names></name><name><surname>Compton</surname><given-names>P</given-names></name><article-title>Improving open information extraction for informal web documents with ripple-down rules</article-title><source>Knowledge Management and Acquisition for Intelligent Systems</source><year>2012</year><publisher-name>Heidelberg: Springer</publisher-name><fpage>160</fpage><lpage>174</lpage></mixed-citation></ref><ref id="B109"><mixed-citation publication-type="other"><collab>OvidSP</collab><comment><ext-link ext-link-type="uri" xlink:href="http://gateway.ovid.com">http://gateway.ovid.com</ext-link></comment></mixed-citation></ref><ref id="B110"><mixed-citation publication-type="other"><collab>PubMed Central</collab><comment><ext-link ext-link-type="uri" xlink:href="http://www.ncbi.nlm.nih.gov/pmc">http://www.ncbi.nlm.nih.gov/pmc</ext-link></comment></mixed-citation></ref><ref id="B111"><mixed-citation publication-type="other"><collab>Google Scholar</collab><comment><ext-link ext-link-type="uri" xlink:href="http://scholar.google.com">http://scholar.google.com</ext-link></comment></mixed-citation></ref><ref id="B112"><mixed-citation publication-type="other"><collab>Microsoft Academic Search</collab><comment><ext-link ext-link-type="uri" xlink:href="http://academic.research.microsoft.com">http://academic.research.microsoft.com</ext-link></comment></mixed-citation></ref><ref id="B113"><mixed-citation publication-type="journal"><name><surname>Rodriguez-Esteban</surname><given-names>R</given-names></name><name><surname>Iossifov</surname><given-names>I</given-names></name><article-title>Figure mining for biomedical research</article-title><source>Bioinformatics</source><year>2009</year><volume>25</volume><fpage>2082</fpage><lpage>2084</lpage><pub-id pub-id-type="pmid">19439564</pub-id></mixed-citation></ref><ref id="B114"><mixed-citation publication-type="book"><name><surname>Ceri</surname><given-names>S</given-names></name><name><surname>Bozzon</surname><given-names>A</given-names></name><name><surname>Brambilla</surname><given-names>M</given-names></name><name><surname>Della Valle</surname><given-names>E</given-names></name><name><surname>Fraternali</surname><given-names>P</given-names></name><name><surname>Quarteroni</surname><given-names>S</given-names></name><article-title>The Information Retrieval Process</article-title><source>Web Information Retrieval</source><year>2013</year><publisher-name>Heidelberg: Springer</publisher-name><fpage>13</fpage><lpage>26</lpage></mixed-citation></ref><ref id="B115"><mixed-citation publication-type="journal"><name><surname>Horsley</surname><given-names>T</given-names></name><name><surname>Dingwall</surname><given-names>O</given-names></name><name><surname>Sampson</surname><given-names>M</given-names></name><article-title>Checking reference lists to find additional studies for systematic reviews</article-title><source>Cochrane Database Syst Rev</source><year>2011</year><volume>8</volume><fpage>1</fpage><lpage>27</lpage></mixed-citation></ref><ref id="B116"><mixed-citation publication-type="journal"><name><surname>Robinson</surname><given-names>KA</given-names></name><name><surname>Dunn</surname><given-names>A</given-names></name><name><surname>Tsafnat</surname><given-names>G</given-names></name><name><surname>Glasziou</surname><given-names>P</given-names></name><article-title>Citation networks of trials: feasibility of iterative bidirectional citation searching</article-title><source>J Clin Epidemiol</source><year>2014</year><volume>67</volume><fpage>793</fpage><lpage>799</lpage><pub-id pub-id-type="pmid">24725642</pub-id></mixed-citation></ref><ref id="B117"><mixed-citation publication-type="book"><name><surname>Summerscales</surname><given-names>R</given-names></name><name><surname>Argamon</surname><given-names>S</given-names></name><name><surname>Hupert</surname><given-names>J</given-names></name><name><surname>Schwartz</surname><given-names>A</given-names></name><article-title>Identifying treatments, groups, and outcomes in medical abstracts</article-title><source>The Sixth Midwest Computational Linguistics Colloquium (MCLC 2009)</source><year>2009</year><publisher-name>Bloomington, IN, USA: Indiana University</publisher-name></mixed-citation></ref><ref id="B118"><mixed-citation publication-type="book"><name><surname>Summerscales</surname><given-names>RL</given-names></name><name><surname>Argamon</surname><given-names>S</given-names></name><name><surname>Bai</surname><given-names>S</given-names></name><name><surname>Huperff</surname><given-names>J</given-names></name><name><surname>Schwartzff</surname><given-names>A</given-names></name><article-title>Automatic summarization of results from clinical trials</article-title><source>the 2011 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)</source><year>2011</year><publisher-name>Atlanta, GA, USA</publisher-name><fpage>372</fpage><lpage>377</lpage></mixed-citation></ref><ref id="B119"><mixed-citation publication-type="journal"><name><surname>Hsu</surname><given-names>W</given-names></name><name><surname>Speier</surname><given-names>W</given-names></name><name><surname>Taira</surname><given-names>RK</given-names></name><article-title>Automated extraction of reported statistical analyses: towards a logical representation of clinical trial literature</article-title><source>AMIA Annu Symp Proc</source><year>2012</year><volume>2012</volume><fpage>350</fpage><lpage>359</lpage><pub-id pub-id-type="pmid">23304305</pub-id></mixed-citation></ref><ref id="B120"><mixed-citation publication-type="other"><name><surname>Huwaldt</surname><given-names>JA</given-names></name><source>Plot Digitizer</source><year>2009</year><comment><ext-link ext-link-type="uri" xlink:href="http://plotdigitizer.sf.net">http://plotdigitizer.sf.net</ext-link></comment></mixed-citation></ref><ref id="B121"><mixed-citation publication-type="other"><name><surname>Egner</surname><given-names>J</given-names></name><name><surname>Mitchell</surname><given-names>M</given-names></name><name><surname>Richter</surname><given-names>T</given-names></name><name><surname>Winchen</surname><given-names>T</given-names></name><source>Engauge Digitizer: convert graphs or map files into numbers</source><year>2013</year><comment><ext-link ext-link-type="uri" xlink:href="http://digitizer.sf.net/">http://digitizer.sf.net/</ext-link></comment></mixed-citation></ref><ref id="B122"><mixed-citation publication-type="other"><name><surname>Rohatgi</surname><given-names>A</given-names></name><source>WebPlotDigitizer</source><year>2013</year><comment><ext-link ext-link-type="uri" xlink:href="http://arohatgi.info/WebPlotDigitizer/">http://arohatgi.info/WebPlotDigitizer/</ext-link></comment></mixed-citation></ref><ref id="B123"><mixed-citation publication-type="journal"><name><surname>Imai</surname><given-names>E</given-names></name><name><surname>Chan</surname><given-names>JC</given-names></name><name><surname>Ito</surname><given-names>S</given-names></name><name><surname>Yamasaki</surname><given-names>T</given-names></name><name><surname>Kobayashi</surname><given-names>F</given-names></name><name><surname>Haneda</surname><given-names>M</given-names></name><name><surname>Makino</surname><given-names>H</given-names></name><article-title>Effects of olmesartan on renal and cardiovascular outcomes in type 2 diabetes with overt nephropathy: a multicentre, randomised, placebo-controlled study</article-title><source>Diabetologia</source><year>2011</year><volume>54</volume><fpage>2978</fpage><lpage>2986</lpage><pub-id pub-id-type="pmid">21993710</pub-id></mixed-citation></ref><ref id="B124"><mixed-citation publication-type="journal"><name><surname>Qahtani</surname><given-names>NH</given-names></name><name><surname>Hajeri</surname><given-names>FA</given-names></name><article-title>The effect of hyoscine butylbromide in shortening the first stage of labor: a double blind, randomized, controlled, clinical trial</article-title><source>Ther Clin Risk Manag</source><year>2011</year><volume>7</volume><fpage>495</fpage><lpage>500</lpage><pub-id pub-id-type="pmid">22241946</pub-id></mixed-citation></ref><ref id="B125"><mixed-citation publication-type="journal"><name><surname>Chung</surname><given-names>GY</given-names></name><name><surname>Coiera</surname><given-names>E</given-names></name><article-title>Are decision trees a feasible knowledge representation to guide extraction of critical information from randomized controlled trial reports?</article-title><source>BMC Med Inform Decis Mak</source><year>2008</year><volume>8</volume><fpage>48</fpage><pub-id pub-id-type="pmid">18957129</pub-id></mixed-citation></ref><ref id="B126"><mixed-citation publication-type="other"><collab>Cochrane Register of Studies</collab><comment><ext-link ext-link-type="uri" xlink:href="http://www.metaxis.com/CRSSoftwarePortal/Index.asp">http://www.metaxis.com/CRSSoftwarePortal/Index.asp</ext-link></comment></mixed-citation></ref><ref id="B127"><mixed-citation publication-type="book"><name><surname>Richard</surname><given-names>J</given-names></name><source>Summing Up: the Science of Reviewing Research</source><year>1984</year><publisher-name>Cambridge, MA, USA: Harvard University Press</publisher-name></mixed-citation></ref><ref id="B128"><mixed-citation publication-type="book"><name><surname>Hedges</surname><given-names>LV</given-names></name><name><surname>Olkin</surname><given-names>I</given-names></name><name><surname>Statistiker</surname><given-names>M</given-names></name><source>Statistical Methods for Meta-analysis</source><year>1985</year><publisher-name>New York: Academic Press</publisher-name></mixed-citation></ref><ref id="B129"><mixed-citation publication-type="journal"><name><surname>Lewis</surname><given-names>S</given-names></name><name><surname>Clarke</surname><given-names>M</given-names></name><article-title>Forest plots: trying to see the wood and the trees</article-title><source>BMJ</source><year>2001</year><volume>322</volume><fpage>1479</fpage><pub-id pub-id-type="pmid">11408310</pub-id></mixed-citation></ref><ref id="B130"><mixed-citation publication-type="journal"><name><surname>Egger</surname><given-names>M</given-names></name><name><surname>Smith</surname><given-names>GD</given-names></name><name><surname>Schneider</surname><given-names>M</given-names></name><name><surname>Minder</surname><given-names>C</given-names></name><article-title>Bias in meta-analysis detected by a simple, graphical test</article-title><source>BMJ</source><year>1997</year><volume>315</volume><fpage>629</fpage><lpage>634</lpage><pub-id pub-id-type="pmid">9310563</pub-id></mixed-citation></ref><ref id="B131"><mixed-citation publication-type="journal"><name><surname>Wallace</surname><given-names>BC</given-names></name><name><surname>Schmid</surname><given-names>CH</given-names></name><name><surname>Lau</surname><given-names>J</given-names></name><name><surname>Trikalinos</surname><given-names>TA</given-names></name><article-title>Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data</article-title><source>BMC Med Res Methodol</source><year>2009</year><volume>9</volume><fpage>80</fpage><pub-id pub-id-type="pmid">19961608</pub-id></mixed-citation></ref><ref id="B132"><mixed-citation publication-type="journal"><name><surname>Zamora</surname><given-names>J</given-names></name><name><surname>Abraira</surname><given-names>V</given-names></name><name><surname>Muriel</surname><given-names>A</given-names></name><name><surname>Khan</surname><given-names>K</given-names></name><name><surname>Coomarasamy</surname><given-names>A</given-names></name><article-title>Meta-DiSc: a software for meta-analysis of test accuracy data</article-title><source>BMC Med Res Methodol</source><year>2006</year><volume>6</volume><fpage>31</fpage><pub-id pub-id-type="pmid">16836745</pub-id></mixed-citation></ref><ref id="B133"><mixed-citation publication-type="book"><name><surname>Rosenberg</surname><given-names>MS</given-names></name><name><surname>Adams</surname><given-names>DC</given-names></name><name><surname>Gurevitch</surname><given-names>J</given-names></name><source>MetaWin: statistical software for meta-analysis</source><year>2000</year><publisher-name>Massachusetts, USA: Sinauer Associates Sunderland</publisher-name></mixed-citation></ref><ref id="B134"><mixed-citation publication-type="journal"><name><surname>Bax</surname><given-names>L</given-names></name><name><surname>Yu</surname><given-names>L-M</given-names></name><name><surname>Ikeda</surname><given-names>N</given-names></name><name><surname>Moons</surname><given-names>KG</given-names></name><article-title>A systematic comparison of software dedicated to meta-analysis of causal studies</article-title><source>BMC Med Res Methodol</source><year>2007</year><volume>7</volume><fpage>40</fpage><pub-id pub-id-type="pmid">17845719</pub-id></mixed-citation></ref><ref id="B135"><mixed-citation publication-type="journal"><name><surname>Moher</surname><given-names>D</given-names></name><name><surname>Liberati</surname><given-names>A</given-names></name><name><surname>Tetzlaff</surname><given-names>J</given-names></name><name><surname>Altman</surname><given-names>DG</given-names></name><article-title>Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement</article-title><source>Ann Intern Med</source><year>2009</year><volume>151</volume><fpage>264</fpage><lpage>269</lpage><pub-id pub-id-type="pmid">19622511</pub-id></mixed-citation></ref><ref id="B136"><mixed-citation publication-type="other"><collab>RevMan-HAL</collab><comment><ext-link ext-link-type="uri" xlink:href="http://szg.cochrane.org/revman-hal">http://szg.cochrane.org/revman-hal</ext-link></comment></mixed-citation></ref><ref id="B137"><mixed-citation publication-type="other"><collab>PRISMA flow diagram generator</collab><comment><ext-link ext-link-type="uri" xlink:href="http://old_theta.phm.utoronto.ca/tools/prisma">http://old_theta.phm.utoronto.ca/tools/prisma</ext-link></comment></mixed-citation></ref><ref id="B138"><mixed-citation publication-type="book"><name><surname>Reiter</surname><given-names>E</given-names></name><person-group person-group-type="editor">Clark A, Fox C, Lappin S</person-group><article-title>Natural Language Generation</article-title><source>The Handbook of Computational Linguistics and Natural Language Processing</source><year>2010</year><publisher-name>Oxford, UK: Wiley-Blackwell</publisher-name></mixed-citation></ref><ref id="B139"><mixed-citation publication-type="journal"><name><surname>Elliott</surname><given-names>JH</given-names></name><name><surname>Turner</surname><given-names>T</given-names></name><name><surname>Clavisi</surname><given-names>O</given-names></name><name><surname>Thomas</surname><given-names>J</given-names></name><name><surname>Higgins</surname><given-names>JP</given-names></name><name><surname>Mavergames</surname><given-names>C</given-names></name><name><surname>Gruen</surname><given-names>RL</given-names></name><article-title>Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap</article-title><source>PLoS Med</source><year>2014</year><volume>11</volume><fpage>e1001603</fpage><pub-id pub-id-type="pmid">24558353</pub-id></mixed-citation></ref><ref id="B140"><mixed-citation publication-type="journal"><name><surname>Kawamoto</surname><given-names>K</given-names></name><name><surname>Honey</surname><given-names>A</given-names></name><name><surname>Rubin</surname><given-names>K</given-names></name><article-title>The HL7-OMG healthcare services specification project: motivation, methodology, and deliverables for enabling a semantically interoperable service-oriented architecture for healthcare</article-title><source>J Am Med Inform Assoc</source><year>2009</year><volume>16</volume><fpage>874</fpage><lpage>881</lpage><pub-id pub-id-type="pmid">19717796</pub-id></mixed-citation></ref></ref-list></back></pmc><affiliations><list><country><li>Australie</li></country><settlement><li>Sydney</li></settlement></list><tree><country name="Australie"><noRegion><name sortKey="Tsafnat, Guy" sort="Tsafnat, Guy" uniqKey="Tsafnat G" first="Guy" last="Tsafnat">Guy Tsafnat</name></noRegion><name sortKey="Choong, Miew Keen" sort="Choong, Miew Keen" uniqKey="Choong M" first="Miew Keen" last="Choong">Miew Keen Choong</name><name sortKey="Coiera, Enrico" sort="Coiera, Enrico" uniqKey="Coiera E" first="Enrico" last="Coiera">Enrico Coiera</name><name sortKey="Dunn, Adam" sort="Dunn, Adam" uniqKey="Dunn A" first="Adam" last="Dunn">Adam Dunn</name><name sortKey="Galgani, Filippo" sort="Galgani, Filippo" uniqKey="Galgani F" first="Filippo" last="Galgani">Filippo Galgani</name><name sortKey="Glasziou, Paul" sort="Glasziou, Paul" uniqKey="Glasziou P" first="Paul" last="Glasziou">Paul Glasziou</name></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/OcrV1/Data/Pmc/Checkpoint
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000037 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Checkpoint/biblio.hfd -nk 000037 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Ticri/CIDE
|area= OcrV1
|flux= Pmc
|étape= Checkpoint
|type= RBID
|clé= PMC:4100748
|texte= Systematic review automation technologies
}}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Checkpoint/RBID.i -Sk "pubmed:25005128" \
| HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Checkpoint/biblio.hfd \
| NlmPubMed2Wicri -a OcrV1
| This area was generated with Dilib version V0.6.32. |  |