Annotated Chemical Patent Corpus: A Gold Standard for Text Mining
Identifieur interne : 000057 ( Pmc/Checkpoint ); précédent : 000056; suivant : 000058Annotated Chemical Patent Corpus: A Gold Standard for Text Mining
Auteurs : Saber A. Akhondi [Pays-Bas] ; Alexander G. Klenner [Allemagne] ; Christian Tyrchan [Suède] ; Anil K. Manchala [Inde] ; Kiran Boppana [Inde] ; Daniel Lowe [Royaume-Uni] ; Marc Zimmermann [Allemagne] ; Sarma A. R. P. Jagarlapudi [Inde] ; Roger Sayle [Royaume-Uni] ; Jan A. Kors [Pays-Bas] ; Sorel Muresan [Suède]Source :
- PLoS ONE [ 1932-6203 ] ; 2014.
Abstract
Exploring the chemical and biological space covered by patent applications is crucial in early-stage medicinal chemistry activities. Patent analysis can provide understanding of compound prior art, novelty checking, validation of biological assays, and identification of new starting points for chemical exploration. Extracting chemical and biological entities from patents through manual extraction by expert curators can take substantial amount of time and resources. Text mining methods can help to ease this process. To validate the performance of such methods, a manually annotated patent corpus is essential. In this study we have produced a large gold standard chemical patent corpus. We developed annotation guidelines and selected 200 full patents from the World Intellectual Property Organization, United States Patent and Trademark Office, and European Patent Office. The patents were pre-annotated automatically and made available to four independent annotator groups each consisting of two to ten annotators. The annotators marked chemicals in different subclasses, diseases, targets, and modes of action. Spelling mistakes and spurious line break due to optical character recognition errors were also annotated. A subset of 47 patents was annotated by at least three annotator groups, from which harmonized annotations and inter-annotator agreement scores were derived. One group annotated the full set. The patent corpus includes 400,125 annotations for the full set and 36,537 annotations for the harmonized set. All patents and annotated entities are publicly available at
Url:
DOI: 10.1371/journal.pone.0107477
PubMed: 25268232
PubMed Central: 4182036
Affiliations:
- Allemagne, Inde, Pays-Bas, Royaume-Uni, Suède
- Angleterre, District de Cologne, Hollande-Méridionale, Rhénanie-du-Nord-Westphalie
- Rotterdam, Sankt Augustin
Links toward previous steps (curation, corpus...)
Links to Exploration step
PMC:4182036Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Annotated Chemical Patent Corpus: A Gold Standard for Text Mining</title>
<author><name sortKey="Akhondi, Saber A" sort="Akhondi, Saber A" uniqKey="Akhondi S" first="Saber A." last="Akhondi">Saber A. Akhondi</name>
<affiliation wicri:level="3"><nlm:aff id="aff1"><addr-line>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam, The Netherlands</addr-line>
</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam</wicri:regionArea>
<placeName><settlement type="city">Rotterdam</settlement>
<region nuts="2" type="province">Hollande-Méridionale</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Klenner, Alexander G" sort="Klenner, Alexander G" uniqKey="Klenner A" first="Alexander G." last="Klenner">Alexander G. Klenner</name>
<affiliation wicri:level="3"><nlm:aff id="aff2"><addr-line>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin, Germany</addr-line>
</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin</wicri:regionArea>
<placeName><region type="land" nuts="1">Rhénanie-du-Nord-Westphalie</region>
<region type="district" nuts="2">District de Cologne</region>
<settlement type="city">Sankt Augustin</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Tyrchan, Christian" sort="Tyrchan, Christian" uniqKey="Tyrchan C" first="Christian" last="Tyrchan">Christian Tyrchan</name>
<affiliation wicri:level="1"><nlm:aff id="aff3"><addr-line>RIA Medicinal Chemistry, AstraZeneca R&D Mölndal, Mölndal, Sweden</addr-line>
</nlm:aff>
<country xml:lang="fr">Suède</country>
<wicri:regionArea>RIA Medicinal Chemistry, AstraZeneca R&D Mölndal, Mölndal</wicri:regionArea>
<wicri:noRegion>Mölndal</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Manchala, Anil K" sort="Manchala, Anil K" uniqKey="Manchala A" first="Anil K." last="Manchala">Anil K. Manchala</name>
<affiliation wicri:level="1"><nlm:aff id="aff4"><addr-line>GVK Biosciences Private Limited, Hyderabad, India</addr-line>
</nlm:aff>
<country xml:lang="fr">Inde</country>
<wicri:regionArea>GVK Biosciences Private Limited, Hyderabad</wicri:regionArea>
<wicri:noRegion>Hyderabad</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Boppana, Kiran" sort="Boppana, Kiran" uniqKey="Boppana K" first="Kiran" last="Boppana">Kiran Boppana</name>
<affiliation wicri:level="1"><nlm:aff id="aff4"><addr-line>GVK Biosciences Private Limited, Hyderabad, India</addr-line>
</nlm:aff>
<country xml:lang="fr">Inde</country>
<wicri:regionArea>GVK Biosciences Private Limited, Hyderabad</wicri:regionArea>
<wicri:noRegion>Hyderabad</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Lowe, Daniel" sort="Lowe, Daniel" uniqKey="Lowe D" first="Daniel" last="Lowe">Daniel Lowe</name>
<affiliation wicri:level="2"><nlm:aff id="aff5"><addr-line>NextMove Software Ltd, Cambridge, England</addr-line>
</nlm:aff>
<country>Royaume-Uni</country>
<placeName><region type="country">Angleterre</region>
</placeName>
<wicri:cityArea>NextMove Software Ltd, Cambridge</wicri:cityArea>
</affiliation>
</author>
<author><name sortKey="Zimmermann, Marc" sort="Zimmermann, Marc" uniqKey="Zimmermann M" first="Marc" last="Zimmermann">Marc Zimmermann</name>
<affiliation wicri:level="3"><nlm:aff id="aff6"><addr-line>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin, Germany</addr-line>
</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin</wicri:regionArea>
<placeName><region type="land" nuts="1">Rhénanie-du-Nord-Westphalie</region>
<region type="district" nuts="2">District de Cologne</region>
<settlement type="city">Sankt Augustin</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Jagarlapudi, Sarma A R P" sort="Jagarlapudi, Sarma A R P" uniqKey="Jagarlapudi S" first="Sarma A. R. P." last="Jagarlapudi">Sarma A. R. P. Jagarlapudi</name>
<affiliation wicri:level="1"><nlm:aff id="aff4"><addr-line>GVK Biosciences Private Limited, Hyderabad, India</addr-line>
</nlm:aff>
<country xml:lang="fr">Inde</country>
<wicri:regionArea>GVK Biosciences Private Limited, Hyderabad</wicri:regionArea>
<wicri:noRegion>Hyderabad</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Sayle, Roger" sort="Sayle, Roger" uniqKey="Sayle R" first="Roger" last="Sayle">Roger Sayle</name>
<affiliation wicri:level="2"><nlm:aff id="aff5"><addr-line>NextMove Software Ltd, Cambridge, England</addr-line>
</nlm:aff>
<country>Royaume-Uni</country>
<placeName><region type="country">Angleterre</region>
</placeName>
<wicri:cityArea>NextMove Software Ltd, Cambridge</wicri:cityArea>
</affiliation>
</author>
<author><name sortKey="Kors, Jan A" sort="Kors, Jan A" uniqKey="Kors J" first="Jan A." last="Kors">Jan A. Kors</name>
<affiliation wicri:level="3"><nlm:aff id="aff1"><addr-line>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam, The Netherlands</addr-line>
</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam</wicri:regionArea>
<placeName><settlement type="city">Rotterdam</settlement>
<region nuts="2" type="province">Hollande-Méridionale</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Muresan, Sorel" sort="Muresan, Sorel" uniqKey="Muresan S" first="Sorel" last="Muresan">Sorel Muresan</name>
<affiliation wicri:level="1"><nlm:aff id="aff7"><addr-line>Chemistry Innovation Centre, AstraZeneca R&D Mölndal, Mölndal, Sweden</addr-line>
</nlm:aff>
<country xml:lang="fr">Suède</country>
<wicri:regionArea>Chemistry Innovation Centre, AstraZeneca R&D Mölndal, Mölndal</wicri:regionArea>
<wicri:noRegion>Mölndal</wicri:noRegion>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PMC</idno>
<idno type="pmid">25268232</idno>
<idno type="pmc">4182036</idno>
<idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4182036</idno>
<idno type="RBID">PMC:4182036</idno>
<idno type="doi">10.1371/journal.pone.0107477</idno>
<date when="2014">2014</date>
<idno type="wicri:Area/Pmc/Corpus">000014</idno>
<idno type="wicri:Area/Pmc/Curation">000014</idno>
<idno type="wicri:Area/Pmc/Checkpoint">000057</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Annotated Chemical Patent Corpus: A Gold Standard for Text Mining</title>
<author><name sortKey="Akhondi, Saber A" sort="Akhondi, Saber A" uniqKey="Akhondi S" first="Saber A." last="Akhondi">Saber A. Akhondi</name>
<affiliation wicri:level="3"><nlm:aff id="aff1"><addr-line>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam, The Netherlands</addr-line>
</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam</wicri:regionArea>
<placeName><settlement type="city">Rotterdam</settlement>
<region nuts="2" type="province">Hollande-Méridionale</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Klenner, Alexander G" sort="Klenner, Alexander G" uniqKey="Klenner A" first="Alexander G." last="Klenner">Alexander G. Klenner</name>
<affiliation wicri:level="3"><nlm:aff id="aff2"><addr-line>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin, Germany</addr-line>
</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin</wicri:regionArea>
<placeName><region type="land" nuts="1">Rhénanie-du-Nord-Westphalie</region>
<region type="district" nuts="2">District de Cologne</region>
<settlement type="city">Sankt Augustin</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Tyrchan, Christian" sort="Tyrchan, Christian" uniqKey="Tyrchan C" first="Christian" last="Tyrchan">Christian Tyrchan</name>
<affiliation wicri:level="1"><nlm:aff id="aff3"><addr-line>RIA Medicinal Chemistry, AstraZeneca R&D Mölndal, Mölndal, Sweden</addr-line>
</nlm:aff>
<country xml:lang="fr">Suède</country>
<wicri:regionArea>RIA Medicinal Chemistry, AstraZeneca R&D Mölndal, Mölndal</wicri:regionArea>
<wicri:noRegion>Mölndal</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Manchala, Anil K" sort="Manchala, Anil K" uniqKey="Manchala A" first="Anil K." last="Manchala">Anil K. Manchala</name>
<affiliation wicri:level="1"><nlm:aff id="aff4"><addr-line>GVK Biosciences Private Limited, Hyderabad, India</addr-line>
</nlm:aff>
<country xml:lang="fr">Inde</country>
<wicri:regionArea>GVK Biosciences Private Limited, Hyderabad</wicri:regionArea>
<wicri:noRegion>Hyderabad</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Boppana, Kiran" sort="Boppana, Kiran" uniqKey="Boppana K" first="Kiran" last="Boppana">Kiran Boppana</name>
<affiliation wicri:level="1"><nlm:aff id="aff4"><addr-line>GVK Biosciences Private Limited, Hyderabad, India</addr-line>
</nlm:aff>
<country xml:lang="fr">Inde</country>
<wicri:regionArea>GVK Biosciences Private Limited, Hyderabad</wicri:regionArea>
<wicri:noRegion>Hyderabad</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Lowe, Daniel" sort="Lowe, Daniel" uniqKey="Lowe D" first="Daniel" last="Lowe">Daniel Lowe</name>
<affiliation wicri:level="2"><nlm:aff id="aff5"><addr-line>NextMove Software Ltd, Cambridge, England</addr-line>
</nlm:aff>
<country>Royaume-Uni</country>
<placeName><region type="country">Angleterre</region>
</placeName>
<wicri:cityArea>NextMove Software Ltd, Cambridge</wicri:cityArea>
</affiliation>
</author>
<author><name sortKey="Zimmermann, Marc" sort="Zimmermann, Marc" uniqKey="Zimmermann M" first="Marc" last="Zimmermann">Marc Zimmermann</name>
<affiliation wicri:level="3"><nlm:aff id="aff6"><addr-line>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin, Germany</addr-line>
</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin</wicri:regionArea>
<placeName><region type="land" nuts="1">Rhénanie-du-Nord-Westphalie</region>
<region type="district" nuts="2">District de Cologne</region>
<settlement type="city">Sankt Augustin</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Jagarlapudi, Sarma A R P" sort="Jagarlapudi, Sarma A R P" uniqKey="Jagarlapudi S" first="Sarma A. R. P." last="Jagarlapudi">Sarma A. R. P. Jagarlapudi</name>
<affiliation wicri:level="1"><nlm:aff id="aff4"><addr-line>GVK Biosciences Private Limited, Hyderabad, India</addr-line>
</nlm:aff>
<country xml:lang="fr">Inde</country>
<wicri:regionArea>GVK Biosciences Private Limited, Hyderabad</wicri:regionArea>
<wicri:noRegion>Hyderabad</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Sayle, Roger" sort="Sayle, Roger" uniqKey="Sayle R" first="Roger" last="Sayle">Roger Sayle</name>
<affiliation wicri:level="2"><nlm:aff id="aff5"><addr-line>NextMove Software Ltd, Cambridge, England</addr-line>
</nlm:aff>
<country>Royaume-Uni</country>
<placeName><region type="country">Angleterre</region>
</placeName>
<wicri:cityArea>NextMove Software Ltd, Cambridge</wicri:cityArea>
</affiliation>
</author>
<author><name sortKey="Kors, Jan A" sort="Kors, Jan A" uniqKey="Kors J" first="Jan A." last="Kors">Jan A. Kors</name>
<affiliation wicri:level="3"><nlm:aff id="aff1"><addr-line>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam, The Netherlands</addr-line>
</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam</wicri:regionArea>
<placeName><settlement type="city">Rotterdam</settlement>
<region nuts="2" type="province">Hollande-Méridionale</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Muresan, Sorel" sort="Muresan, Sorel" uniqKey="Muresan S" first="Sorel" last="Muresan">Sorel Muresan</name>
<affiliation wicri:level="1"><nlm:aff id="aff7"><addr-line>Chemistry Innovation Centre, AstraZeneca R&D Mölndal, Mölndal, Sweden</addr-line>
</nlm:aff>
<country xml:lang="fr">Suède</country>
<wicri:regionArea>Chemistry Innovation Centre, AstraZeneca R&D Mölndal, Mölndal</wicri:regionArea>
<wicri:noRegion>Mölndal</wicri:noRegion>
</affiliation>
</author>
</analytic>
<series><title level="j">PLoS ONE</title>
<idno type="eISSN">1932-6203</idno>
<imprint><date when="2014">2014</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en"><p>Exploring the chemical and biological space covered by patent applications is crucial in early-stage medicinal chemistry activities. Patent analysis can provide understanding of compound prior art, novelty checking, validation of biological assays, and identification of new starting points for chemical exploration. Extracting chemical and biological entities from patents through manual extraction by expert curators can take substantial amount of time and resources. Text mining methods can help to ease this process. To validate the performance of such methods, a manually annotated patent corpus is essential. In this study we have produced a large gold standard chemical patent corpus. We developed annotation guidelines and selected 200 full patents from the World Intellectual Property Organization, United States Patent and Trademark Office, and European Patent Office. The patents were pre-annotated automatically and made available to four independent annotator groups each consisting of two to ten annotators. The annotators marked chemicals in different subclasses, diseases, targets, and modes of action. Spelling mistakes and spurious line break due to optical character recognition errors were also annotated. A subset of 47 patents was annotated by at least three annotator groups, from which harmonized annotations and inter-annotator agreement scores were derived. One group annotated the full set. The patent corpus includes 400,125 annotations for the full set and 36,537 annotations for the harmonized set. All patents and annotated entities are publicly available at <ext-link ext-link-type="uri" xlink:href="http://www.biosemantics.org">www.biosemantics.org</ext-link>
.</p>
</div>
</front>
<back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Muresan, S" uniqKey="Muresan S">S Muresan</name>
</author>
<author><name sortKey="Petrov, P" uniqKey="Petrov P">P Petrov</name>
</author>
<author><name sortKey="Southan, C" uniqKey="Southan C">C Southan</name>
</author>
<author><name sortKey="Kjellberg, Mj" uniqKey="Kjellberg M">MJ Kjellberg</name>
</author>
<author><name sortKey="Kogej, T" uniqKey="Kogej T">T Kogej</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Southan, C" uniqKey="Southan C">C Southan</name>
</author>
<author><name sortKey="Boppana, K" uniqKey="Boppana K">K Boppana</name>
</author>
<author><name sortKey="Jagarlapudi, Sa" uniqKey="Jagarlapudi S">SA Jagarlapudi</name>
</author>
<author><name sortKey="Muresan, S" uniqKey="Muresan S">S Muresan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Tyrchan, C" uniqKey="Tyrchan C">C Tyrchan</name>
</author>
<author><name sortKey="Bostrom, J" uniqKey="Bostrom J">J Boström</name>
</author>
<author><name sortKey="Giordanetto, F" uniqKey="Giordanetto F">F Giordanetto</name>
</author>
<author><name sortKey="Winter, J" uniqKey="Winter J">J Winter</name>
</author>
<author><name sortKey="Muresan, S" uniqKey="Muresan S">S Muresan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Kolarik, C" uniqKey="Kolarik C">C Kolarik</name>
</author>
<author><name sortKey="Hofmann Apitius, M" uniqKey="Hofmann Apitius M">M Hofmann-Apitius</name>
</author>
<author><name sortKey="Zimmermann, M" uniqKey="Zimmermann M">M Zimmermann</name>
</author>
<author><name sortKey="Fluck, J" uniqKey="Fluck J">J Fluck</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Klinger, R" uniqKey="Klinger R">R Klinger</name>
</author>
<author><name sortKey="Kolarik, C" uniqKey="Kolarik C">C Kolarik</name>
</author>
<author><name sortKey="Fluck, J" uniqKey="Fluck J">J Fluck</name>
</author>
<author><name sortKey="Hofmann Apitius, M" uniqKey="Hofmann Apitius M">M Hofmann-Apitius</name>
</author>
<author><name sortKey="Friedrich, Cm" uniqKey="Friedrich C">CM Friedrich</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Zimmermann, M" uniqKey="Zimmermann M">M Zimmermann</name>
</author>
<author><name sortKey="Fluck, J" uniqKey="Fluck J">J Fluck</name>
</author>
<author><name sortKey="Thi, Lt" uniqKey="Thi L">LT Thi</name>
</author>
<author><name sortKey="Kolarik, C" uniqKey="Kolarik C">C Kolarik</name>
</author>
<author><name sortKey="Kumpf, K" uniqKey="Kumpf K">K Kumpf</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Tseng, Y H" uniqKey="Tseng Y">Y-H Tseng</name>
</author>
<author><name sortKey="Lin, C J" uniqKey="Lin C">C-J Lin</name>
</author>
<author><name sortKey="Lin, Y I" uniqKey="Lin Y">Y-I Lin</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Jessop, Dm" uniqKey="Jessop D">DM Jessop</name>
</author>
<author><name sortKey="Adams, Se" uniqKey="Adams S">SE Adams</name>
</author>
<author><name sortKey="Murray Rust, P" uniqKey="Murray Rust P">P Murray-Rust</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Kiss, M" uniqKey="Kiss M">M Kiss</name>
</author>
<author><name sortKey="Nagy, A" uniqKey="Nagy A">Á Nagy</name>
</author>
<author><name sortKey="Vincze, V" uniqKey="Vincze V">V Vincze</name>
</author>
<author><name sortKey="Almasi, A" uniqKey="Almasi A">A Almási</name>
</author>
<author><name sortKey="Alexin, Z" uniqKey="Alexin Z">Z Alexin</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Vazquez, M" uniqKey="Vazquez M">M Vazquez</name>
</author>
<author><name sortKey="Krallinger, M" uniqKey="Krallinger M">M Krallinger</name>
</author>
<author><name sortKey="Leitner, F" uniqKey="Leitner F">F Leitner</name>
</author>
<author><name sortKey="Valencia, A" uniqKey="Valencia A">A Valencia</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Kim, Jd" uniqKey="Kim J">JD Kim</name>
</author>
<author><name sortKey="Ohta, T" uniqKey="Ohta T">T Ohta</name>
</author>
<author><name sortKey="Tateisi, Y" uniqKey="Tateisi Y">Y Tateisi</name>
</author>
<author><name sortKey="Tsujii, J" uniqKey="Tsujii J">J Tsujii</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Kulick, S" uniqKey="Kulick S">S Kulick</name>
</author>
<author><name sortKey="Bies, A" uniqKey="Bies A">A Bies</name>
</author>
<author><name sortKey="Liberman, M" uniqKey="Liberman M">M Liberman</name>
</author>
<author><name sortKey="Mandel, M" uniqKey="Mandel M">M Mandel</name>
</author>
<author><name sortKey="Mcdonald, R" uniqKey="Mcdonald R">R McDonald</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Kolarik, C" uniqKey="Kolarik C">C Kolárik</name>
</author>
<author><name sortKey="Klinger, R" uniqKey="Klinger R">R Klinger</name>
</author>
<author><name sortKey="Friedrich, Cm" uniqKey="Friedrich C">CM Friedrich</name>
</author>
<author><name sortKey="Hofmann Apitius, M" uniqKey="Hofmann Apitius M">M Hofmann-Apitius</name>
</author>
<author><name sortKey="Fluck, J" uniqKey="Fluck J">J Fluck</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Krallinger, M" uniqKey="Krallinger M">M Krallinger</name>
</author>
<author><name sortKey="Leitner, F" uniqKey="Leitner F">F Leitner</name>
</author>
<author><name sortKey="Rabal, O" uniqKey="Rabal O">O Rabal</name>
</author>
<author><name sortKey="Vazquez, M" uniqKey="Vazquez M">M Vazquez</name>
</author>
<author><name sortKey="Oyarzabal, J" uniqKey="Oyarzabal J">J Oyarzabal</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Grego, T" uniqKey="Grego T">T Grego</name>
</author>
<author><name sortKey="P Zik, P" uniqKey="P Zik P">P Pęzik</name>
</author>
<author><name sortKey="Couto, Fm" uniqKey="Couto F">FM Couto</name>
</author>
<author><name sortKey="Rebholz Schuhmann, D" uniqKey="Rebholz Schuhmann D">D Rebholz-Schuhmann</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Degtyarenko, K" uniqKey="Degtyarenko K">K Degtyarenko</name>
</author>
<author><name sortKey="De Matos, P" uniqKey="De Matos P">P De Matos</name>
</author>
<author><name sortKey="Ennis, M" uniqKey="Ennis M">M Ennis</name>
</author>
<author><name sortKey="Hastings, J" uniqKey="Hastings J">J Hastings</name>
</author>
<author><name sortKey="Zbinden, M" uniqKey="Zbinden M">M Zbinden</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="De Matos, P" uniqKey="De Matos P">P De Matos</name>
</author>
<author><name sortKey="Alcantara, R" uniqKey="Alcantara R">R Alcántara</name>
</author>
<author><name sortKey="Dekker, A" uniqKey="Dekker A">A Dekker</name>
</author>
<author><name sortKey="Ennis, M" uniqKey="Ennis M">M Ennis</name>
</author>
<author><name sortKey="Hastings, J" uniqKey="Hastings J">J Hastings</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Sayle, R" uniqKey="Sayle R">R Sayle</name>
</author>
<author><name sortKey="Xie, Ph" uniqKey="Xie P">PH Xie</name>
</author>
<author><name sortKey="Muresan, S" uniqKey="Muresan S">S Muresan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Akhondi, Sa" uniqKey="Akhondi S">SA Akhondi</name>
</author>
<author><name sortKey="Kors, Ja" uniqKey="Kors J">JA Kors</name>
</author>
<author><name sortKey="Muresan, S" uniqKey="Muresan S">S Muresan</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Weininger, D" uniqKey="Weininger D">D Weininger</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Heller, S" uniqKey="Heller S">S Heller</name>
</author>
<author><name sortKey="Mcnaught, A" uniqKey="Mcnaught A">A McNaught</name>
</author>
<author><name sortKey="Stein, S" uniqKey="Stein S">S Stein</name>
</author>
<author><name sortKey="Tchekhovskoi, D" uniqKey="Tchekhovskoi D">D Tchekhovskoi</name>
</author>
<author><name sortKey="Pletnev, I" uniqKey="Pletnev I">I Pletnev</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Lowe, Dm" uniqKey="Lowe D">DM Lowe</name>
</author>
<author><name sortKey="Sayle, Ra" uniqKey="Sayle R">RA Sayle</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Stenetorp, P" uniqKey="Stenetorp P">P Stenetorp</name>
</author>
<author><name sortKey="Pyysalo, S" uniqKey="Pyysalo S">S Pyysalo</name>
</author>
<author><name sortKey="Topi, G" uniqKey="Topi G">G Topić</name>
</author>
<author><name sortKey="Ohta, T" uniqKey="Ohta T">T Ohta</name>
</author>
<author><name sortKey="Ananiadou, S" uniqKey="Ananiadou S">S Ananiadou</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Lewin, I" uniqKey="Lewin I">I Lewin</name>
</author>
<author><name sortKey="Kafkas, S" uniqKey="Kafkas S">S Kafkas</name>
</author>
<author><name sortKey="Rebholz Schuhmann, D" uniqKey="Rebholz Schuhmann D">D Rebholz-Schuhmann</name>
</author>
</analytic>
</biblStruct>
</listBibl>
</div1>
</back>
</TEI>
<pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">PLoS One</journal-id>
<journal-id journal-id-type="iso-abbrev">PLoS ONE</journal-id>
<journal-id journal-id-type="publisher-id">plos</journal-id>
<journal-id journal-id-type="pmc">plosone</journal-id>
<journal-title-group><journal-title>PLoS ONE</journal-title>
</journal-title-group>
<issn pub-type="epub">1932-6203</issn>
<publisher><publisher-name>Public Library of Science</publisher-name>
<publisher-loc>San Francisco, USA</publisher-loc>
</publisher>
</journal-meta>
<article-meta><article-id pub-id-type="pmid">25268232</article-id>
<article-id pub-id-type="pmc">4182036</article-id>
<article-id pub-id-type="publisher-id">PONE-D-14-17575</article-id>
<article-id pub-id-type="doi">10.1371/journal.pone.0107477</article-id>
<article-categories><subj-group subj-group-type="heading"><subject>Research Article</subject>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Computer and Information Sciences</subject>
<subj-group><subject>Information Technology</subject>
<subj-group><subject>Text Mining</subject>
</subj-group>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Medicine and Health Sciences</subject>
<subj-group><subject>Pharmaceutics</subject>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Research and Analysis Methods</subject>
<subj-group><subject>Database and Informatics Methods</subject>
<subj-group><subject>Bioinformatics</subject>
</subj-group>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Social Sciences</subject>
<subj-group><subject>Law and Legal Sciences</subject>
<subj-group><subject>Commercial Law</subject>
<subj-group><subject>Patents</subject>
</subj-group>
</subj-group>
</subj-group>
</subj-group>
</article-categories>
<title-group><article-title>Annotated Chemical Patent Corpus: A Gold Standard for Text Mining</article-title>
<alt-title alt-title-type="running-head">Annotated Chemical Patent Corpus: A Gold Standard for Text Mining</alt-title>
</title-group>
<contrib-group><contrib contrib-type="author"><name><surname>Akhondi</surname>
<given-names>Saber A.</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Klenner</surname>
<given-names>Alexander G.</given-names>
</name>
<xref ref-type="aff" rid="aff2"><sup>2</sup>
</xref>
<xref ref-type="author-notes" rid="fn1"><sup>¤a</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Tyrchan</surname>
<given-names>Christian</given-names>
</name>
<xref ref-type="aff" rid="aff3"><sup>3</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Manchala</surname>
<given-names>Anil K.</given-names>
</name>
<xref ref-type="aff" rid="aff4"><sup>4</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Boppana</surname>
<given-names>Kiran</given-names>
</name>
<xref ref-type="aff" rid="aff4"><sup>4</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Lowe</surname>
<given-names>Daniel</given-names>
</name>
<xref ref-type="aff" rid="aff5"><sup>5</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Zimmermann</surname>
<given-names>Marc</given-names>
</name>
<xref ref-type="aff" rid="aff6"><sup>6</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Jagarlapudi</surname>
<given-names>Sarma A. R. P.</given-names>
</name>
<xref ref-type="aff" rid="aff4"><sup>4</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Sayle</surname>
<given-names>Roger</given-names>
</name>
<xref ref-type="aff" rid="aff5"><sup>5</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Kors</surname>
<given-names>Jan A.</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
<xref ref-type="corresp" rid="cor1"><sup>*</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Muresan</surname>
<given-names>Sorel</given-names>
</name>
<xref ref-type="aff" rid="aff7"><sup>7</sup>
</xref>
<xref ref-type="corresp" rid="cor1"><sup>*</sup>
</xref>
<xref ref-type="author-notes" rid="fn2"><sup>¤b</sup>
</xref>
</contrib>
</contrib-group>
<aff id="aff1"><label>1</label>
<addr-line>Department of Medical Informatics, Erasmus University Medical Centre, Rotterdam, The Netherlands</addr-line>
</aff>
<aff id="aff2"><label>2</label>
<addr-line>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin, Germany</addr-line>
</aff>
<aff id="aff3"><label>3</label>
<addr-line>RIA Medicinal Chemistry, AstraZeneca R&D Mölndal, Mölndal, Sweden</addr-line>
</aff>
<aff id="aff4"><label>4</label>
<addr-line>GVK Biosciences Private Limited, Hyderabad, India</addr-line>
</aff>
<aff id="aff5"><label>5</label>
<addr-line>NextMove Software Ltd, Cambridge, England</addr-line>
</aff>
<aff id="aff6"><label>6</label>
<addr-line>Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI), Fraunhofer-Gesellschaft, Sankt Augustin, Germany</addr-line>
</aff>
<aff id="aff7"><label>7</label>
<addr-line>Chemistry Innovation Centre, AstraZeneca R&D Mölndal, Mölndal, Sweden</addr-line>
</aff>
<contrib-group><contrib contrib-type="editor"><name><surname>Ranganathan</surname>
<given-names>Shoba</given-names>
</name>
<role>Editor</role>
<xref ref-type="aff" rid="edit1"></xref>
</contrib>
</contrib-group>
<aff id="edit1"><addr-line>Macquarie University, Australia</addr-line>
</aff>
<author-notes><corresp id="cor1">* E-mail: <email>j.kors@erasmusmc.nl</email>
(JAK); <email>sorel.muresan@gmail.com</email>
(SM)</corresp>
<fn fn-type="conflict"><p><bold>Competing Interests: </bold>
AstraZeneca is a global biopharmaceutical company specialising in the discovery, development, manufacturing and marketing of prescription medicines. At the time of the study SM and CT were affiliated to AstraZeneca. NextMove Software Ltd. develops and sells commercial software for text mining. RS and DL were affiliated to NextMove Software Ltd. GVK Biosciences Private Limited is a discovery research and development organization. GVK BIO provides services across the R&D and manufacturing value chain. SARPJ, AKM and KB were affiliated to GVK BIO. The authors confirm that the above statement does not alter their adherence to all PLOS ONE policies on sharing data and materials, as detailed online in guide for authors.</p>
</fn>
<fn fn-type="con"><p>Conceived and designed the experiments: SAA AGK CT MZ JAK SM. Performed the experiments: SAA AGK CT AKM KB DL MZ SARPJ RS SM. Analyzed the data: SAA CT JAK SM. Wrote the paper: SAA JAK SM.</p>
</fn>
<fn id="fn1" fn-type="current-aff"><label>¤a</label>
<p>Current address: European Patent Office, Rijswijk, The Netherlands</p>
</fn>
<fn id="fn2" fn-type="current-aff"><label>¤b</label>
<p>Current address: AkzoNobel Surface Chemistry AB, Stenungsund, Sweden</p>
</fn>
</author-notes>
<pub-date pub-type="collection"><year>2014</year>
</pub-date>
<pub-date pub-type="epub"><day>30</day>
<month>9</month>
<year>2014</year>
</pub-date>
<volume>9</volume>
<issue>9</issue>
<elocation-id>e107477</elocation-id>
<history><date date-type="received"><day>22</day>
<month>4</month>
<year>2014</year>
</date>
<date date-type="accepted"><day>10</day>
<month>8</month>
<year>2014</year>
</date>
</history>
<permissions><copyright-year>2014</copyright-year>
<copyright-holder>Akhondi et al</copyright-holder>
<license><license-p>This is an open-access article distributed under the terms of the <ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution License</ext-link>
, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.</license-p>
</license>
</permissions>
<abstract><p>Exploring the chemical and biological space covered by patent applications is crucial in early-stage medicinal chemistry activities. Patent analysis can provide understanding of compound prior art, novelty checking, validation of biological assays, and identification of new starting points for chemical exploration. Extracting chemical and biological entities from patents through manual extraction by expert curators can take substantial amount of time and resources. Text mining methods can help to ease this process. To validate the performance of such methods, a manually annotated patent corpus is essential. In this study we have produced a large gold standard chemical patent corpus. We developed annotation guidelines and selected 200 full patents from the World Intellectual Property Organization, United States Patent and Trademark Office, and European Patent Office. The patents were pre-annotated automatically and made available to four independent annotator groups each consisting of two to ten annotators. The annotators marked chemicals in different subclasses, diseases, targets, and modes of action. Spelling mistakes and spurious line break due to optical character recognition errors were also annotated. A subset of 47 patents was annotated by at least three annotator groups, from which harmonized annotations and inter-annotator agreement scores were derived. One group annotated the full set. The patent corpus includes 400,125 annotations for the full set and 36,537 annotations for the harmonized set. All patents and annotated entities are publicly available at <ext-link ext-link-type="uri" xlink:href="http://www.biosemantics.org">www.biosemantics.org</ext-link>
.</p>
</abstract>
<funding-group><funding-statement>This study was made possible by a studentship grant provided by AstraZeneca for SAA. The funder provided support in the form of salaries for SAA, but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.</funding-statement>
</funding-group>
<counts><page-count count="8"></page-count>
</counts>
<custom-meta-group><custom-meta id="data-availability"><meta-name>Data Availability</meta-name>
<meta-value>The authors confirm that all data underlying the findings are fully available without restriction. All patents and annotated entities are publicly available at <ext-link ext-link-type="uri" xlink:href="http://www.biosemantics.org">www.biosemantics.org</ext-link>
.</meta-value>
</custom-meta>
</custom-meta-group>
</article-meta>
<notes><title>Data Availability</title>
<p>The authors confirm that all data underlying the findings are fully available without restriction. All patents and annotated entities are publicly available at <ext-link ext-link-type="uri" xlink:href="http://www.biosemantics.org">www.biosemantics.org</ext-link>
.</p>
</notes>
</front>
<body><sec id="s1"><title>Introduction</title>
<p>A substantial number of patent applications are filed every year by the pharmaceutical sector <xref rid="pone.0107477-Muresan1" ref-type="bibr">[1]</xref>
. Exploring the chemical and biological space covered by these patents is crucial in early-stage medicinal chemistry activities <xref rid="pone.0107477-Muresan1" ref-type="bibr">[1]</xref>
, <xref rid="pone.0107477-Southan1" ref-type="bibr">[2]</xref>
. Patent specifications are one of many information sources needed to progress drug discovery projects. Patent analysis can provide understanding of compound prior art, novelty checking, validation of biological assays, and identification of new starting points for chemical exploration <xref rid="pone.0107477-Tyrchan1" ref-type="bibr">[3]</xref>
.</p>
<p>Extracting chemical and biological entities from patents is a complex task <xref rid="pone.0107477-Kolarik1" ref-type="bibr">[4]</xref>
, <xref rid="pone.0107477-Klinger1" ref-type="bibr">[5]</xref>
. Different approaches are currently used including manual extraction by expert curators, text mining supported by chemical and biological named entity recognition, or combinations thereof <xref rid="pone.0107477-Zimmermann1" ref-type="bibr">[6]</xref>
. Chemical patents are complex legal documents that can contain up to hundreds of pages. The European Patent Office (EPO) <xref rid="pone.0107477-European1" ref-type="bibr">[7]</xref>
, the pharmaceutically relevant patents within the United States Patent and Trademark Office (USPTO) <xref rid="pone.0107477-United1" ref-type="bibr">[8]</xref>
, and the World Intellectual Property Organization (WIPO) <xref rid="pone.0107477-World1" ref-type="bibr">[9]</xref>
can be accessed and queried on-line via their websites. The patents are freely available from the patent offices, usually as XML, HTML or image PDFs, although EPO limits the number of downloads per week for non-paying users. Using optical character recognition (OCR), the image PDFs can be prepared for text mining. In fact, the available HTML and XML documents are mainly the OCR output prepared and published by the patent offices.</p>
<p>However, the text mining itself is a rather challenging task <xref rid="pone.0107477-Tseng1" ref-type="bibr">[10]</xref>
, <xref rid="pone.0107477-Jessop1" ref-type="bibr">[11]</xref>
. Methods and their output can suffer dramatically from the large number of complex chemical names, term ambiguities, complex syntactic structures and OCR errors <xref rid="pone.0107477-Kiss1" ref-type="bibr">[12]</xref>
.</p>
<p>To validate the performance of named entity recognition techniques, the availability of a manually annotated patent corpus is essential <xref rid="pone.0107477-Vazquez1" ref-type="bibr">[13]</xref>
. Producing such annotated text is laborious and expensive. Most of the prior focus on corpora development has been on genes and proteins and less effort has been put into creating corpora for chemical terms <xref rid="pone.0107477-Corbett1" ref-type="bibr">[14]</xref>
. Among the latter efforts, Kim et al. <xref rid="pone.0107477-Kim1" ref-type="bibr">[15]</xref>
in 2003 developed the GENIA corpus consisting of several classes of chemicals. The BioIE corpus by Kulick et al. <xref rid="pone.0107477-Kulick1" ref-type="bibr">[16]</xref>
was made available in 2004 and included annotations of chemicals and proteins. In 2008, Kolárik et al. <xref rid="pone.0107477-Kolrik1" ref-type="bibr">[17]</xref>
released a small corpus of scientific abstracts annotated with chemical compounds. Recently, the CHEMDNER corpus, annotated with different classes of chemicals, was made available as part of the BioCreative challenge <xref rid="pone.0107477-Krallinger1" ref-type="bibr">[18]</xref>
. All these corpora consist of scientific abstracts from Medline. In a collaborative project between the EPO and the Chemical Entities of Biological Interest (ChEBI) in 2009 <xref rid="pone.0107477-Grego1" ref-type="bibr">[19]</xref>
a chemical patent corpus containing annotations of chemical entities and, if possible, their mapping to ChEBI chemical compounds <xref rid="pone.0107477-Degtyarenko1" ref-type="bibr">[20]</xref>
was developed. In a later study <xref rid="pone.0107477-Tiago1" ref-type="bibr">[21]</xref>
, the updated version of ChEBI <xref rid="pone.0107477-DeMatos1" ref-type="bibr">[22]</xref>
was used to increase the number of mappings. A larger patent corpus was developed in 2012 by Kiss et al. <xref rid="pone.0107477-Kiss1" ref-type="bibr">[12]</xref>
which included name entity recognition of generic chemical compounds.</p>
<p>To our knowledge, the development of a gold standard patent corpus has not been systematically tackled before. Among the obvious reasons for this are the length and complexity of the patent text. In previous attempts only limited number of chemicals have been annotated and subclasses have not been defined. Other biological entities such as diseases or modes of actions have not been included and errors due to misspellings or OCR procedures have not been considered. Most previous studies on annotated corpora did not provide insights into inter-annotator agreement. This information would be valuable in assessing and comparing the performance of text mining applications.</p>
<p>Here we present a gold standard annotated corpus of 200 full patents for benchmarking text mining performance. The patent corpus includes annotation of chemicals with subclasses, diseases, targets and modes of action. Also spelling mistakes and spurious line break due to OCR errors are annotated within this corpus. The full-text patents and annotated entities are publicly available at <ext-link ext-link-type="uri" xlink:href="http://www.biosemantics.org">www.biosemantics.org</ext-link>
.</p>
</sec>
<sec sec-type="materials|methods" id="s2"><title>Methods and Materials</title>
<sec id="s2a"><title>Corpus development strategy</title>
<p>The development of the gold standard patent corpus consisted of several phases. First, annotation guidelines were developed and a set of 200 diverse patents was chosen. The patents were pre-annotated automatically and made available to four independent annotator groups. The annotator groups could choose to consider or disregard the pre-annotations. Two patents were used to refine the annotation guidelines. The remaining patents were distributed between multiple annotator groups in a way that a subset of 47 patents was annotated by at least three groups, from which harmonized annotations were derived. Inter-annotator agreement scores between the annotator groups and against the harmonized set were computed. One annotator group annotated the complete set of patents.</p>
</sec>
<sec id="s2b"><title>Patent corpus selection</title>
<p>The GVK BIO target class database <xref rid="pone.0107477-GVK1" ref-type="bibr">[23]</xref>
was used as a starting point for patent corpus selection. Patents from the EPO <xref rid="pone.0107477-European1" ref-type="bibr">[7]</xref>
, USPTO <xref rid="pone.0107477-United1" ref-type="bibr">[8]</xref>
, and WIPO <xref rid="pone.0107477-World1" ref-type="bibr">[9]</xref>
are available through this database, which includes relationships between documents, assays, chemical structures, assignees and protein targets, manually abstracted by expert curators <xref rid="pone.0107477-Muresan1" ref-type="bibr">[1]</xref>
. Within the database, patents are binned based on different classes of protein families such as kinases or GPCRs <xref rid="pone.0107477-GVK1" ref-type="bibr">[23]</xref>
.</p>
<p>All English language patents containing between 10 and 200 exemplified compounds, with a named primary target, were selected from the GVK BIO database. We made sure that all compounds had a molecular weight below 1000 to bias towards small-molecule patents. We did not specify limits on the time of the application. Overall 28,695 patents fulfilled the above criteria.</p>
<p>Chemical patents are known to include long sentences with complex syntactic structure <xref rid="pone.0107477-Kiss1" ref-type="bibr">[12]</xref>
. Individual companies may have different ways of writing patents and we wanted to include diversity over assignees in the corpora. Therefore, if assignees had written multiple patents for one primary target, only one was randomly kept and the rest was disregarded.</p>
<p>Based on these selection criteria we were left with 8,016 patents grouped in 11 target classes. To make sure that a collection of well-known patents are included in the corpus, 50 drug patents from Sayle et al. <xref rid="pone.0107477-Sayle1" ref-type="bibr">[24]</xref>
were added. Subsequently patents were randomly picked from each target group with a minimum of 10 patents per group. The diversity of the final selection is shown in <xref ref-type="table" rid="pone-0107477-t001">Table 1</xref>
. The final set consists of 121 USPTO, 66 WIPO, and 13 EPO patents, and contains over 11,500 pages and 4.2 million words.</p>
<table-wrap id="pone-0107477-t001" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0107477.t001</object-id>
<label>Table 1</label>
<caption><title>Target class distribution of the 8,066 patents from which the final set was drawn.</title>
</caption>
<alternatives><graphic id="pone-0107477-t001-1" xlink:href="pone.0107477.t001"></graphic>
<table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
</colgroup>
<thead><tr><td align="left" rowspan="1" colspan="1">Target class</td>
<td align="left" rowspan="1" colspan="1">Number of patents</td>
<td align="left" rowspan="1" colspan="1">Final selection</td>
</tr>
</thead>
<tbody><tr><td align="left" rowspan="1" colspan="1">GPCR</td>
<td align="left" rowspan="1" colspan="1">3,569</td>
<td align="left" rowspan="1" colspan="1">20</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Protease</td>
<td align="left" rowspan="1" colspan="1">1,093</td>
<td align="left" rowspan="1" colspan="1">17</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Kinase</td>
<td align="left" rowspan="1" colspan="1">1,046</td>
<td align="left" rowspan="1" colspan="1">12</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Ion-Channel</td>
<td align="left" rowspan="1" colspan="1">433</td>
<td align="left" rowspan="1" colspan="1">14</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Oxidoreductase</td>
<td align="left" rowspan="1" colspan="1">404</td>
<td align="left" rowspan="1" colspan="1">17</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Hydrolase</td>
<td align="left" rowspan="1" colspan="1">364</td>
<td align="left" rowspan="1" colspan="1">15</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">NHR</td>
<td align="left" rowspan="1" colspan="1">349</td>
<td align="left" rowspan="1" colspan="1">15</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Transporters</td>
<td align="left" rowspan="1" colspan="1">323</td>
<td align="left" rowspan="1" colspan="1">18</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Other</td>
<td align="left" rowspan="1" colspan="1">218</td>
<td align="left" rowspan="1" colspan="1">11</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Transferase</td>
<td align="left" rowspan="1" colspan="1">152</td>
<td align="left" rowspan="1" colspan="1">12</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Phosphatase</td>
<td align="left" rowspan="1" colspan="1">65</td>
<td align="left" rowspan="1" colspan="1">17</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Drugs from Sayle et al. <xref rid="pone.0107477-Sayle1" ref-type="bibr">[24]</xref>
</td>
<td align="left" rowspan="1" colspan="1">50</td>
<td align="left" rowspan="1" colspan="1">32</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Total</td>
<td align="left" rowspan="1" colspan="1">8,066</td>
<td align="left" rowspan="1" colspan="1">200</td>
</tr>
</tbody>
</table>
</alternatives>
</table-wrap>
<p>The patents were downloaded from the sources (EPO, USPTO, and WIPO) in XML format. Whenever multiple consecutive line breaks were encountered, they were replaced with a single line break. Images were also removed for all patents.</p>
</sec>
<sec id="s2c"><title>Annotated entities</title>
<p>We annotated all compounds, diseases, protein targets, and modes of actions (MOA) mentioned in the patents. Compounds were assigned to a number of subclasses based on how they are generated: systematic identifiers and non-systematic identifiers <xref rid="pone.0107477-Akhondi1" ref-type="bibr">[25]</xref>
. The following systematic identifiers were annotated: IUPAC names <xref rid="pone.0107477-About1" ref-type="bibr">[26]</xref>
, such as “ammonium phosphate” or “2-[2-(4-{2-[ethyl(2-fluorobenzyl)amino]-2-oxoethoxy}phenyl)ethoxy]benzoic acid”; SMILES notations <xref rid="pone.0107477-Weininger1" ref-type="bibr">[27]</xref>
, such as “n1c[nH]cc1”; and InChI strings <xref rid="pone.0107477-Heller1" ref-type="bibr">[28]</xref>
, <xref rid="pone.0107477-About2" ref-type="bibr">[29]</xref>
, such as “InChI = 1S/C2H6O/c1-2-3/h3H,2H2,1H3”. We also annotated the following non-systematic identifiers: trademarks, such as “Aspirin”, “Mesupron”, and “Arimidex”; abbreviations, such as “DCM”, “TBTU” and “DMAP”; CAS numbers <xref rid="pone.0107477-Undefined1" ref-type="bibr">[30]</xref>
, <xref rid="pone.0107477-CAS1" ref-type="bibr">[31]</xref>
, such as “7732-18-5”; formulas, such as “MgSO4”; registry numbers, such as “ly256548”; and generic names, such as “iodotamoxifen”, “cycloalkylamines” and “racemate”. Any mention of diseases, such as “diabetes”, protein targets, such as “trypsin”, and MOAs, such as “antagonist”, were also annotated. OCR errors were also annotated in terms of spelling mistakes and spurious line breaks.</p>
</sec>
<sec id="s2d"><title>Annotation guidelines</title>
<p>Initial annotation guidelines were developed based on previous work <xref rid="pone.0107477-Corbett1" ref-type="bibr">[14]</xref>
, <xref rid="pone.0107477-Kulick1" ref-type="bibr">[16]</xref>
–<xref rid="pone.0107477-Krallinger1" ref-type="bibr">[18]</xref>
. Two patents (US5023269 <xref rid="pone.0107477-Krushinski1" ref-type="bibr">[32]</xref>
and US4659716 <xref rid="pone.0107477-Villani1" ref-type="bibr">[33]</xref>
) were randomly chosen from the patent corpus for training the annotators and fine-tuning the annotation guidelines. The following rules were defined:</p>
<list list-type="order"><list-item><p>When an entity is nested or has an overlap with another entity, annotate the entity that is more specific and informative. For example “5-HT1D” should not be annotated as target when it is embedded within the target annotation of “5-HT1D Serotonin Receptors”.</p>
</list-item>
<list-item><p>Annotate simple IUPAC names such as “water, “ammonia”, and “ethanol”.</p>
</list-item>
<list-item><p>Prefixes should be included within annotations, for example “1,4-” in “1,4-butanediol”.</p>
</list-item>
<list-item><p>Simple formulas such as “NaOH” and “(NH4)2SO4”, should be annotated as Formulas.</p>
</list-item>
<list-item><p>Counterions, such as “acetate”, “oxalate”, “propionate”, should be annotated as IUPAC names.</p>
</list-item>
<list-item><p>Generic structures such as “4-halo-phenol” or “xylene”, should be annotated as Generic names.</p>
</list-item>
<list-item><p>Polymers, e.g., “Polystyrene”, should be annotated as Generic names.</p>
</list-item>
<list-item><p>Trivial names, e.g., “Sildenafil”, should be annotated as IUPAC names.</p>
</list-item>
<list-item><p>Enumerations, like “hydrochloric” and “hydrobromic” in “include inorganic acids such as hydrochloric, hydrobromic”, should be annotated as IUPAC names.</p>
</list-item>
<list-item><p>Elements like “N”, “O”, and “C” should not be annotated.</p>
</list-item>
<list-item><p>Misspelled terms should be annotated as spelling mistakes (e.g., “hydrobroml:c”).</p>
</list-item>
<list-item><p>Annotations spanning over multiple lines because of spurious line breaks should be annotated as one term and be tagged with spurious line breaks.</p>
</list-item>
<list-item><p>Extra white space should be annotated as spelling mistakes (e.g., “hydro bromic”).</p>
</list-item>
<list-item><p>Do not annotate a term if it is splitted due to reasons other than OCR errors.</p>
</list-item>
<list-item><p>All symbols such as comma, charge symbol or brackets, should be included in the annotation (e.g., “n1c[nH]cc1”).</p>
</list-item>
</list>
</sec>
<sec id="s2e"><title>Annotation process</title>
<p>Each patent was automatically pre-annotated using LeadMine (NextMove Software, UK) <xref rid="pone.0107477-Lowe1" ref-type="bibr">[34]</xref>
. LeadMine can identify chemicals, protein targets, genes, species, company names, and also has the ability to recognise terms with spelling mistakes and suggest corrections. This increases the likelihood of detecting terms with OCR errors by the human annotator.</p>
<p>A pre-annotation consists of the span of text corresponding with the entity and its location within the text file. The following entity types were pre-annotated by LeadMine: IUPAC names, trivial names, CAS numbers, registry numbers, generic names, formulas, and targets. We did not pre-annotate SMILES and InChIs, as they are rarely present in patents. Diseases, and MOAs were also not included as this was not possible through our version of LeadMine.</p>
<p>For the annotation process the Brat rapid annotation tool (version 1.3) was used <xref rid="pone.0107477-Stenetorp1" ref-type="bibr">[35]</xref>
. Brat allows online annotation of text using pre-defined entity types. It can display the pre-annotations and annotators can add new annotations and modify or delete the pre-annotated entities. To reduce mistakes and increase readability each entity type was marked by a specific color. For performance reasons we split the patents into pages with 50 paragraphs for display in Brat. <xref ref-type="fig" rid="pone-0107477-g001">Figure 1</xref>
shows a screenshot of Brat with pre-annotations.</p>
<fig id="pone-0107477-g001" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0107477.g001</object-id>
<label>Figure 1</label>
<caption><p>Example patent text with pre-annotations as shown by the Brat annotation tool.</p>
</caption>
<graphic xlink:href="pone.0107477.g001"></graphic>
</fig>
<p>Patents were annotated by annotators from four groups: AstraZeneca, Fraunhofer, GVK BIO, and NextMove. The GVK BIO annotation group consisted of ten annotators, while the other annotator groups had two annotators. One annotator group (Fraunhofer) chose to disregard the pre-annotations made by LeadMine. The patents were distributed between annotators within a group, such that each patent was annotated by only one annotator in a group. In the context of this paper, annotator group will refer to any individual annotator within the group.</p>
<p>Annotators had to correct any misidentified pre-annotation and had to add annotations that were missed in the pre-annotation step. Entities containing misspellings or spurious line breaks were separately annotated.</p>
</sec>
<sec id="s2f"><title>Resolving misannotation of ambiguous terms</title>
<p>After the completion of the annotations by all groups, a group of annotators reviewed the results to reduce the number of ambiguous terms within the corpus. A term is defined as ambiguous if different groups annotated it with different entity types throughout the corpus.</p>
<p>A list of ambiguously annotated terms was compiled and annotators were asked to review the list only based on the different entity types assigned to each ambiguous term (i.e., the context of the terms was not provided). The annotators had to classify each term in one of three groups:</p>
<list list-type="simple"><list-item><p>1- None of the entity types assigned to the term is applicable. All annotations of the term were removed from the corpus. For example, “nitrogen” was annotated as both IUPAC and Generic multiple times throughout the corpus. However, either entity type is incorrect since the term is an element. Therefore annotations of nitrogen are removed from the corpus.</p>
</list-item>
<list-item><p>2- One entity type is applicable. All occurrences of the term within the corpus were assigned to this entity type. For example, the term “DMF” was assigned 43 times as Trademark, 289 times as Abbreviation, and once as Formula. Regardless of the context of the text, DMF is an abbreviation and therefore the entity type of the term was changed to Abbreviation throughout the corpus.</p>
</list-item>
<list-item><p>3- More than one entity type is applicable. Only term annotations with an entity type that is not applicable, were removed throughout the corpus. For example, the term “5-ht” has been annotated 17 times as Abbreviation, 25 times as Generic, and 23 times as Target. Depending on the context of the text, the term can be either Target or Abbreviation but not Generic. Therefore all annotations of the term as Generic were removed from the corpus.</p>
</list-item>
</list>
</sec>
<sec id="s2g"><title>Harmonization</title>
<p>To develop the gold standard corpus, the annotations of the 47 patents annotated by more than three groups were merged into a harmonized set. The centroid algorithm described by Lewin et al. <xref rid="pone.0107477-Lewin1" ref-type="bibr">[36]</xref>
was used for this purpose.</p>
<p>Briefly, the algorithm tokenizes the annotations of different annotators at the character level and counts the number of agreeing annotators over pairs of adjacent annotation-internal characters <xref rid="pone.0107477-Lewin1" ref-type="bibr">[36]</xref>
. Calculating votes over annotation-internal character pairs and not individual characters, guarantees that boundaries (starting and ending position of an annotated entity type) are considered in situations where two terms are annotated directly adjacent to each other <xref rid="pone.0107477-Lewin1" ref-type="bibr">[36]</xref>
. The harmonized annotation consists of the characters pairs that have a vote equal to or larger than a specified threshold. In this work, we used a voting threshold of two, i.e., at least two annotators had to agree on the annotation.</p>
<p>The centroid algorithm was executed separately for each entity type. Therefore votes were only calculated if at least two annotators annotated a term with the same entity type.</p>
</sec>
<sec id="s2h"><title>Inter-annotator agreement</title>
<p>Similar to Corbett et al. <xref rid="pone.0107477-Corbett1" ref-type="bibr">[14]</xref>
and Kolárik et al. <xref rid="pone.0107477-Kolrik1" ref-type="bibr">[17]</xref>
, we used the F-score (harmonic mean of recall and precision) to calculate the inter-annotator agreement between the annotator groups and between each annotator group and the harmonized set. For the comparison of two sets of annotations, one set was arbitrarily chosen as the gold standard (this choice does not affect the F-score). An annotation in the other set was counted as true positive if it was identical to the gold standard annotation, i.e., if both annotations had the same entity type and the same start and end location. If a gold standard annotation was not given, or not rendered exactly in the other set (i.e., non-matching boundaries or a different entity type), it was counted as false negative; if an annotation found in the other set did not exactly match the gold standard, it was counted as false positive.</p>
</sec>
</sec>
<sec id="s3"><title>Results</title>
<sec id="s3a"><title>Patent distribution among groups</title>
<p>The number of annotated patents varied between annotation groups. Apart from the two patents used for training, 27 patents were annotated by NextMove, 36 by Fraunhofer, 49 by AstraZeneca, and 198 by GVK BIO. A total of 47 patents were annotated by at least three of the groups (three patents were annotated by all four groups).</p>
</sec>
<sec id="s3b"><title>Initial harmonized set</title>
<p>The initial harmonized set, prior to disambiguation, was generated over the 47 common patents, yielding a total of 35,337 annotations (<xref ref-type="table" rid="pone-0107477-t002">Table 2</xref>
). The results show that IUPAC names and generic names have been annotated significantly more than any other chemical type, as has also been shown previously <xref rid="pone.0107477-Vazquez1" ref-type="bibr">[13]</xref>
. On the other hand, InChIs, CAS registry numbers and SMILES are rarely seen in these chemical patents. Also, a considerable number of diseases, targets, and MOAs have been annotated.</p>
<table-wrap id="pone-0107477-t002" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0107477.t002</object-id>
<label>Table 2</label>
<caption><title>Number of annotated terms and unique terms within the harmonized set prior to disambiguation.</title>
</caption>
<alternatives><graphic id="pone-0107477-t002-2" xlink:href="pone.0107477.t002"></graphic>
<table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
</colgroup>
<thead><tr><td align="left" rowspan="1" colspan="1">Entity type</td>
<td align="left" rowspan="1" colspan="1">Annotated terms</td>
<td align="left" rowspan="1" colspan="1">Unique terms</td>
</tr>
</thead>
<tbody><tr><td align="left" rowspan="1" colspan="1">IUPAC</td>
<td align="left" rowspan="1" colspan="1">14,423</td>
<td align="left" rowspan="1" colspan="1">5,365</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Generic</td>
<td align="left" rowspan="1" colspan="1">7,959</td>
<td align="left" rowspan="1" colspan="1">880</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Disease</td>
<td align="left" rowspan="1" colspan="1">3,777</td>
<td align="left" rowspan="1" colspan="1">1,257</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Target</td>
<td align="left" rowspan="1" colspan="1">3,227</td>
<td align="left" rowspan="1" colspan="1">705</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Trademark</td>
<td align="left" rowspan="1" colspan="1">2,273</td>
<td align="left" rowspan="1" colspan="1">987</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Abbreviation</td>
<td align="left" rowspan="1" colspan="1">1,460</td>
<td align="left" rowspan="1" colspan="1">153</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Formula</td>
<td align="left" rowspan="1" colspan="1">1,069</td>
<td align="left" rowspan="1" colspan="1">171</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">MOA</td>
<td align="left" rowspan="1" colspan="1">1,014</td>
<td align="left" rowspan="1" colspan="1">211</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Registry Number</td>
<td align="left" rowspan="1" colspan="1">108</td>
<td align="left" rowspan="1" colspan="1">90</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">SMILES</td>
<td align="left" rowspan="1" colspan="1">21</td>
<td align="left" rowspan="1" colspan="1">21</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">CAS</td>
<td align="left" rowspan="1" colspan="1">6</td>
<td align="left" rowspan="1" colspan="1">5</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">InChI</td>
<td align="left" rowspan="1" colspan="1">0</td>
<td align="left" rowspan="1" colspan="1">0</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Total</td>
<td align="left" rowspan="1" colspan="1">35,337</td>
<td align="left" rowspan="1" colspan="1">9,845</td>
</tr>
</tbody>
</table>
</alternatives>
</table-wrap>
</sec>
<sec id="s3c"><title>Inter-annotator agreement prior to disambiguation</title>
<p><xref ref-type="table" rid="pone-0107477-t003">Table 3</xref>
shows the inter-annotator agreement between the groups and the harmonized set prior to disambiguation. There is generally a higher inter-annotator agreement between individual annotator groups and the harmonized set than between pairs of groups. The best agreement was 0.78. The agreement between groups ranged between 0.39 and 0.69. Investigation of the reasons for some low agreements suggested that adding a disambiguation step could resolve some of these disagreements.</p>
<table-wrap id="pone-0107477-t003" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0107477.t003</object-id>
<label>Table 3</label>
<caption><title>Inter-annotator agreement (F-score) without ambiguity resolution.</title>
</caption>
<alternatives><graphic id="pone-0107477-t003-3" xlink:href="pone.0107477.t003"></graphic>
<table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
</colgroup>
<thead><tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">AstraZeneca</td>
<td align="left" rowspan="1" colspan="1">Fraunhofer</td>
<td colspan="2" align="left" rowspan="1">GVK BIO</td>
<td align="left" rowspan="1" colspan="1">NextMove</td>
</tr>
</thead>
<tbody><tr><td align="left" rowspan="1" colspan="1">Fraunhofer</td>
<td align="left" rowspan="1" colspan="1">0.42</td>
<td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1"></td>
<td colspan="2" align="left" rowspan="1"></td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">GVK BIO</td>
<td align="left" rowspan="1" colspan="1">0.60</td>
<td align="left" rowspan="1" colspan="1">0.39</td>
<td align="left" rowspan="1" colspan="1"></td>
<td colspan="2" align="left" rowspan="1"></td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">NextMove</td>
<td align="left" rowspan="1" colspan="1">0.50</td>
<td align="left" rowspan="1" colspan="1">0.69</td>
<td align="left" rowspan="1" colspan="1">0.52</td>
<td colspan="2" align="left" rowspan="1"></td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Harmonized</td>
<td align="left" rowspan="1" colspan="1">0.78</td>
<td align="left" rowspan="1" colspan="1">0.64</td>
<td align="left" rowspan="1" colspan="1">0.74</td>
<td colspan="2" align="left" rowspan="1">0.72</td>
</tr>
</tbody>
</table>
</alternatives>
</table-wrap>
</sec>
<sec id="s3d"><title>Disambiguation</title>
<p>A set of 2,135 unique ambiguous terms, corresponding to 47,044 annotations, were provided to annotators for disambiguation as described above. The annotators were able to make a decision for 333 unique ambiguous terms, affecting 9,005 annotations. The results in <xref ref-type="table" rid="pone-0107477-t004">Table 4</xref>
show that most difficulties within the annotations were encountered between IUPAC names, Generic names and Trademarks. Also 23 elements were found that had been annotated 2,499 times with different entity types throughout the corpus. Since elements should not be annotated according to the guidelines, these terms were removed from the corpus.</p>
<table-wrap id="pone-0107477-t004" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0107477.t004</object-id>
<label>Table 4</label>
<caption><title>The effect of the disambiguation process on the annotations.</title>
</caption>
<alternatives><graphic id="pone-0107477-t004-4" xlink:href="pone.0107477.t004"></graphic>
<table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
</colgroup>
<thead><tr><td align="left" rowspan="1" colspan="1">Rules</td>
<td align="left" rowspan="1" colspan="1">Type</td>
<td align="left" rowspan="1" colspan="1">Affected Terms</td>
<td align="left" rowspan="1" colspan="1">Affected Annotations</td>
</tr>
</thead>
<tbody><tr><td align="left" rowspan="1" colspan="1">Add</td>
<td align="left" rowspan="1" colspan="1">IUPAC</td>
<td align="left" rowspan="1" colspan="1">52</td>
<td align="left" rowspan="1" colspan="1">2,275</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">annotation</td>
<td align="left" rowspan="1" colspan="1">Abbreviation</td>
<td align="left" rowspan="1" colspan="1">29</td>
<td align="left" rowspan="1" colspan="1">1,631</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Generic</td>
<td align="left" rowspan="1" colspan="1">67</td>
<td align="left" rowspan="1" colspan="1">976</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Trademark</td>
<td align="left" rowspan="1" colspan="1">71</td>
<td align="left" rowspan="1" colspan="1">442</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Disease</td>
<td align="left" rowspan="1" colspan="1">4</td>
<td align="left" rowspan="1" colspan="1">387</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">MOA</td>
<td align="left" rowspan="1" colspan="1">2</td>
<td align="left" rowspan="1" colspan="1">203</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Formula</td>
<td align="left" rowspan="1" colspan="1">25</td>
<td align="left" rowspan="1" colspan="1">177</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Registry Number</td>
<td align="left" rowspan="1" colspan="1">28</td>
<td align="left" rowspan="1" colspan="1">111</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Target</td>
<td align="left" rowspan="1" colspan="1">19</td>
<td align="left" rowspan="1" colspan="1">32</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Remove</td>
<td align="left" rowspan="1" colspan="1">Elements</td>
<td align="left" rowspan="1" colspan="1">23</td>
<td align="left" rowspan="1" colspan="1">2,499</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">annotation</td>
<td align="left" rowspan="1" colspan="1">IUPAC</td>
<td align="left" rowspan="1" colspan="1">7</td>
<td align="left" rowspan="1" colspan="1">103</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Trademark</td>
<td align="left" rowspan="1" colspan="1">3</td>
<td align="left" rowspan="1" colspan="1">101</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Generic</td>
<td align="left" rowspan="1" colspan="1">2</td>
<td align="left" rowspan="1" colspan="1">67</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Target</td>
<td align="left" rowspan="1" colspan="1">1</td>
<td align="left" rowspan="1" colspan="1">1</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Total</td>
<td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">333</td>
<td align="left" rowspan="1" colspan="1">9,005</td>
</tr>
</tbody>
</table>
</alternatives>
</table-wrap>
</sec>
<sec id="s3e"><title>Inter-annotator agreement after disambiguation</title>
<p>After resolving the ambiguous terms, the harmonized set was recalculated. This resulted in an increase of inter-annotator agreement scores by 0.01 to 0.09 points (<xref ref-type="table" rid="pone-0107477-t005">Table 5</xref>
).</p>
<table-wrap id="pone-0107477-t005" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0107477.t005</object-id>
<label>Table 5</label>
<caption><title>Inter-annotator agreement after ambiguity resolution.</title>
</caption>
<alternatives><graphic id="pone-0107477-t005-5" xlink:href="pone.0107477.t005"></graphic>
<table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
</colgroup>
<thead><tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">AstraZeneca</td>
<td align="left" rowspan="1" colspan="1">Fraunhofer</td>
<td colspan="2" align="left" rowspan="1">GVK BIO</td>
<td align="left" rowspan="1" colspan="1">NextMove</td>
<td align="left" rowspan="1" colspan="1">Harmonized</td>
</tr>
</thead>
<tbody><tr><td align="left" rowspan="1" colspan="1">AstraZeneca</td>
<td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">+ 0.04</td>
<td align="left" rowspan="1" colspan="1">+ 0.09</td>
<td colspan="2" align="left" rowspan="1">+ 0.08</td>
<td align="left" rowspan="1" colspan="1">+ 0.06</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Fraunhofer</td>
<td align="left" rowspan="1" colspan="1">0.46</td>
<td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">+ 0.05</td>
<td colspan="2" align="left" rowspan="1">+ 0.03</td>
<td align="left" rowspan="1" colspan="1">+ 0.01</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">GVK BIO</td>
<td align="left" rowspan="1" colspan="1">0.69</td>
<td align="left" rowspan="1" colspan="1">0.44</td>
<td align="left" rowspan="1" colspan="1"></td>
<td colspan="2" align="left" rowspan="1">+ 0.06</td>
<td align="left" rowspan="1" colspan="1">+ 0.05</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">NextMove</td>
<td align="left" rowspan="1" colspan="1">0.58</td>
<td align="left" rowspan="1" colspan="1">0.72</td>
<td align="left" rowspan="1" colspan="1">0.58</td>
<td colspan="2" align="left" rowspan="1"></td>
<td align="left" rowspan="1" colspan="1">+ 0.03</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Harmonized</td>
<td align="left" rowspan="1" colspan="1">0.84</td>
<td align="left" rowspan="1" colspan="1">0.65</td>
<td align="left" rowspan="1" colspan="1">0.79</td>
<td colspan="2" align="left" rowspan="1">0.75</td>
<td align="left" rowspan="1" colspan="1"></td>
</tr>
</tbody>
</table>
</alternatives>
<table-wrap-foot><fn id="nt101"><label></label>
<p>The lower left triangle presents the inter-annotator agreement scores (F-score). The upper right triangle shows the improvement gained through disambiguation.</p>
</fn>
</table-wrap-foot>
</table-wrap>
<p>Recalculating the inter-annotator agreement by only considering text boundaries and disregarding the entity types, further increases the agreement with up to 0.04 points. To analyze the reasons behind some of the low agreements, inter-annotator agreement scores were calculated for the main entity types (<xref ref-type="table" rid="pone-0107477-t006">Table 6</xref>
). The major difficulty in the annotation was encountered for non-systematic identifiers and MOAs, while identification of targets, diseases, and systematic identifiers were made with higher agreements.</p>
<table-wrap id="pone-0107477-t006" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0107477.t006</object-id>
<label>Table 6</label>
<caption><title>Inter-annotator agreement (F-score) between the harmonized set and the annotator groups for the main entity types.</title>
</caption>
<alternatives><graphic id="pone-0107477-t006-6" xlink:href="pone.0107477.t006"></graphic>
<table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
</colgroup>
<thead><tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">AstraZeneca Harmonized</td>
<td align="left" rowspan="1" colspan="1">Fraunhofer Harmonized</td>
<td align="left" rowspan="1" colspan="1">GVK BIO Harmonized</td>
<td align="left" rowspan="1" colspan="1">NextMove Harmonized</td>
</tr>
</thead>
<tbody><tr><td align="left" rowspan="1" colspan="1">Overall</td>
<td align="left" rowspan="1" colspan="1">0.84</td>
<td align="left" rowspan="1" colspan="1">0.65</td>
<td align="left" rowspan="1" colspan="1">0.79</td>
<td align="left" rowspan="1" colspan="1">0.75</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Chemicals</td>
<td align="left" rowspan="1" colspan="1">0.89</td>
<td align="left" rowspan="1" colspan="1">0.65</td>
<td align="left" rowspan="1" colspan="1">0.78</td>
<td align="left" rowspan="1" colspan="1">0.75</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Systematic</td>
<td align="left" rowspan="1" colspan="1">0.94</td>
<td align="left" rowspan="1" colspan="1">0.81</td>
<td align="left" rowspan="1" colspan="1">0.91</td>
<td align="left" rowspan="1" colspan="1">0.93</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Non-systematic</td>
<td align="left" rowspan="1" colspan="1">0.85</td>
<td align="left" rowspan="1" colspan="1">0.38</td>
<td align="left" rowspan="1" colspan="1">0.68</td>
<td align="left" rowspan="1" colspan="1">0.56</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Disease</td>
<td align="left" rowspan="1" colspan="1">0.47</td>
<td align="left" rowspan="1" colspan="1">0.82</td>
<td align="left" rowspan="1" colspan="1">0.87</td>
<td align="left" rowspan="1" colspan="1">0.86</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Targets</td>
<td align="left" rowspan="1" colspan="1">0.76</td>
<td align="left" rowspan="1" colspan="1">0.57</td>
<td align="left" rowspan="1" colspan="1">0.81</td>
<td align="left" rowspan="1" colspan="1">0.86</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">MOA</td>
<td align="left" rowspan="1" colspan="1">0.65</td>
<td align="left" rowspan="1" colspan="1">0.29</td>
<td align="left" rowspan="1" colspan="1">0.67</td>
<td align="left" rowspan="1" colspan="1">0.17</td>
</tr>
</tbody>
</table>
</alternatives>
</table-wrap>
<p>The inter-annotator agreement between the groups and overall, chemicals and systematic names were between 0.65 and 0.94. The inter-annotator agreement for non-systematic terms between Fraunhofer and the harmonized set was only 0.38. To investigate the reasons behind this low agreement, we recalculated the inter-annotator agreement between Fraunhofer and the harmonized set by considering cases where one annotation was embedded within the other annotation as an agreement. This only increased the inter-annotator score to 0.46. Further analysis showed that counting annotations that overlap as an agreement increased the score to 0.62. The main reason for the remaining differences was that annotators at Fraunhofer did not annotate formulas and had low agreements with others within the generic terms.</p>
<p><xref ref-type="table" rid="pone-0107477-t006">Table 6</xref>
shows that apart from AstraZeneca, all groups managed to gain a high inter-annotator agreement (0.82 to 0.86) between diseases and the harmonized set. Further analysis showed that the low inter-annotator agreement between AstraZeneca and the harmonized set on diseases is due to annotation differences in the boundaries. Calculating inter-annotator agreement on diseases by also accepting embedded terms increased the agreement to 0.70.</p>
<p>The inter-annotator agreement between Fraunhofer and the harmonized set for targets was only 0.57. Additional investigation showed that accepting embedded terms increased the agreement to 0.64.</p>
<p>The annotations of MOA for Fraunhofer and NextMove were also greatly affected by how the boundaries were chosen. An example is the term “mixed agonist” for which one group annotated the whole term as MOA and the other only annotated “agonist” as MOA. Accepting such cases as an agreement increases the agreement between NextMove and the harmonized set from 0.17 to 0.72, and between Fraunhofer and the harmonized set from 0.29 to 0.62.</p>
</sec>
<sec id="s3f"><title>The gold standard patent corpus</title>
<p>The gold standard patent corpus consists of two sets: the harmonized corpus and the full corpus. The harmonized corpus consists of 47 patents with a total of 36,537 annotations for 9,813 unique terms (<xref ref-type="table" rid="pone-0107477-t007">Table 7</xref>
). In addition, 1,239 OCR errors have been annotated, of which 1,189 are spelling mistakes. The full patent corpus of 198 patents contains only the GVK BIO annotations with 400,125 annotations for 80,977 unique terms. The set includes 5,096 OCR error annotations, of which 4,403 are spelling mistakes.</p>
<table-wrap id="pone-0107477-t007" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0107477.t007</object-id>
<label>Table 7</label>
<caption><title>Number of annotated terms and unique terms in the harmonized set and in the full patent set of the gold standard corpus after disambiguation.</title>
</caption>
<alternatives><graphic id="pone-0107477-t007-7" xlink:href="pone.0107477.t007"></graphic>
<table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
<col align="center" span="1"></col>
</colgroup>
<thead><tr><td align="left" rowspan="1" colspan="1"></td>
<td colspan="2" align="left" rowspan="1">Harmonized set (47 Patents)</td>
<td colspan="2" align="left" rowspan="1">Full set (198 Patents)</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1"></td>
<td align="left" rowspan="1" colspan="1">Unique terms</td>
<td align="left" rowspan="1" colspan="1">Annotated terms</td>
<td align="left" rowspan="1" colspan="1">Unique terms</td>
<td align="left" rowspan="1" colspan="1">Annotated terms</td>
</tr>
</thead>
<tbody><tr><td align="left" rowspan="1" colspan="1">IUPAC</td>
<td align="left" rowspan="1" colspan="1">5,325</td>
<td align="left" rowspan="1" colspan="1">14,377</td>
<td align="left" rowspan="1" colspan="1">50,893</td>
<td align="left" rowspan="1" colspan="1">135,603</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Generic</td>
<td align="left" rowspan="1" colspan="1">881</td>
<td align="left" rowspan="1" colspan="1">8,384</td>
<td align="left" rowspan="1" colspan="1">14,305</td>
<td align="left" rowspan="1" colspan="1">169,133</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Disease</td>
<td align="left" rowspan="1" colspan="1">1,256</td>
<td align="left" rowspan="1" colspan="1">3,776</td>
<td align="left" rowspan="1" colspan="1">4,503</td>
<td align="left" rowspan="1" colspan="1">20,229</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Target</td>
<td align="left" rowspan="1" colspan="1">703</td>
<td align="left" rowspan="1" colspan="1">3,235</td>
<td align="left" rowspan="1" colspan="1">3,514</td>
<td align="left" rowspan="1" colspan="1">14,398</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Trademark</td>
<td align="left" rowspan="1" colspan="1">994</td>
<td align="left" rowspan="1" colspan="1">2,366</td>
<td align="left" rowspan="1" colspan="1">3,365</td>
<td align="left" rowspan="1" colspan="1">9,574</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Abbreviation</td>
<td align="left" rowspan="1" colspan="1">153</td>
<td align="left" rowspan="1" colspan="1">2,088</td>
<td align="left" rowspan="1" colspan="1">778</td>
<td align="left" rowspan="1" colspan="1">21,087</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Formula</td>
<td align="left" rowspan="1" colspan="1">169</td>
<td align="left" rowspan="1" colspan="1">1,127</td>
<td align="left" rowspan="1" colspan="1">3,108</td>
<td align="left" rowspan="1" colspan="1">25,716</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">MOA</td>
<td align="left" rowspan="1" colspan="1">210</td>
<td align="left" rowspan="1" colspan="1">1,017</td>
<td align="left" rowspan="1" colspan="1">110</td>
<td align="left" rowspan="1" colspan="1">3,837</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Registry Number</td>
<td align="left" rowspan="1" colspan="1">96</td>
<td align="left" rowspan="1" colspan="1">140</td>
<td align="left" rowspan="1" colspan="1">188</td>
<td align="left" rowspan="1" colspan="1">329</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">SMILES</td>
<td align="left" rowspan="1" colspan="1">21</td>
<td align="left" rowspan="1" colspan="1">21</td>
<td align="left" rowspan="1" colspan="1">166</td>
<td align="left" rowspan="1" colspan="1">166</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">CAS</td>
<td align="left" rowspan="1" colspan="1">5</td>
<td align="left" rowspan="1" colspan="1">6</td>
<td align="left" rowspan="1" colspan="1">47</td>
<td align="left" rowspan="1" colspan="1">53</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">InChI</td>
<td align="left" rowspan="1" colspan="1">0</td>
<td align="left" rowspan="1" colspan="1">0</td>
<td align="left" rowspan="1" colspan="1">0</td>
<td align="left" rowspan="1" colspan="1">0</td>
</tr>
<tr><td align="left" rowspan="1" colspan="1">Total</td>
<td align="left" rowspan="1" colspan="1">9,813</td>
<td align="left" rowspan="1" colspan="1">36,537</td>
<td align="left" rowspan="1" colspan="1">80,977</td>
<td align="left" rowspan="1" colspan="1">400,125</td>
</tr>
</tbody>
</table>
</alternatives>
</table-wrap>
</sec>
</sec>
<sec id="s4"><title>Discussion and Conclusions</title>
<p>We have produced a gold standard chemical patent corpus consisting of 198 full patents of which 47 patents have been annotated by at least three annotators. The patent corpus contains a selection of patents from WIPO, USPTO and EPO with annotation of compounds, diseases, targets, and MOAs. We have also annotated spelling errors for the mentioned entity types.</p>
<p>We have released the inter-annotator agreements along with the gold standard corpus. Making inter-annotator agreement scores available will hopefully prove to be useful for performance assessment of automatic annotations of the patent corpus.</p>
<p>To our knowledge this is the first patent gold standard corpus containing full patents with different entity types (chemicals and their sub entities, diseases, MOAs, and targets). Patents are one of the richest knowledge sources with high information content and detailed description of chemistry and technology. Our annotation process showed the complexity of the annotation task. The OCR process added a significant level of noise to the text. A high inter-annotator agreement was seen on the annotation of entities such as systematic names. In contrast, we observed lower inter-annotator agreements for non-systematic names and MOAs. This emphasizes the challenges in identifying named entities from patent text. Annotation of OCR errors may also be helpful to improve patent informatics systems by facilitating the development of algorithms to correct such errors.</p>
<p>The annotated gold standard corpus should prove a valuable resource for developing and evaluating patent text analytics approaches.</p>
<sec id="s4a"><title>Availability</title>
<p>The gold standard corpus is available through <ext-link ext-link-type="uri" xlink:href="http://www.biosemantics.org">www.biosemantics.org</ext-link>
.</p>
</sec>
</sec>
</body>
<back><ack><p>The authors would like to acknowledge the patent curation team at GVK BIO for annotating the full patent corpus.</p>
</ack>
<ref-list><title>References</title>
<ref id="pone.0107477-Muresan1"><label>1</label>
<mixed-citation publication-type="journal"><name><surname>Muresan</surname>
<given-names>S</given-names>
</name>
, <name><surname>Petrov</surname>
<given-names>P</given-names>
</name>
, <name><surname>Southan</surname>
<given-names>C</given-names>
</name>
, <name><surname>Kjellberg</surname>
<given-names>MJ</given-names>
</name>
, <name><surname>Kogej</surname>
<given-names>T</given-names>
</name>
, <etal>et al</etal>
(<year>2011</year>
) <article-title>Making every SAR point count: the development of Chemistry Connect for the large-scale integration of structure and bioactivity data</article-title>
. <source>Drug Discov Today</source>
<volume>16</volume>
: <fpage>1019</fpage>
–<lpage>1030</lpage>
.<pub-id pub-id-type="pmid">22024215</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Southan1"><label>2</label>
<mixed-citation publication-type="journal"><name><surname>Southan</surname>
<given-names>C</given-names>
</name>
, <name><surname>Boppana</surname>
<given-names>K</given-names>
</name>
, <name><surname>Jagarlapudi</surname>
<given-names>SA</given-names>
</name>
, <name><surname>Muresan</surname>
<given-names>S</given-names>
</name>
(<year>2011</year>
) <article-title>Analysis of in vitro bioactivity data extracted from drug discovery literature and patents: Ranking 1654 human protein targets by assayed compounds and molecular scaffolds</article-title>
. <source>J Cheminform</source>
<volume>3</volume>
: <fpage>14</fpage>
.<pub-id pub-id-type="pmid">21569515</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Tyrchan1"><label>3</label>
<mixed-citation publication-type="journal"><name><surname>Tyrchan</surname>
<given-names>C</given-names>
</name>
, <name><surname>Boström</surname>
<given-names>J</given-names>
</name>
, <name><surname>Giordanetto</surname>
<given-names>F</given-names>
</name>
, <name><surname>Winter</surname>
<given-names>J</given-names>
</name>
, <name><surname>Muresan</surname>
<given-names>S</given-names>
</name>
(<year>2012</year>
) <article-title>Exploiting Structural Information in Patent Specifications for Key Compound Prediction</article-title>
. <source>J Chem Inf Model</source>
<volume>52</volume>
: <fpage>1480</fpage>
–<lpage>1489</lpage>
.<pub-id pub-id-type="pmid">22639789</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Kolarik1"><label>4</label>
<mixed-citation publication-type="journal"><name><surname>Kolarik</surname>
<given-names>C</given-names>
</name>
, <name><surname>Hofmann-Apitius</surname>
<given-names>M</given-names>
</name>
, <name><surname>Zimmermann</surname>
<given-names>M</given-names>
</name>
, <name><surname>Fluck</surname>
<given-names>J</given-names>
</name>
(<year>2007</year>
) <article-title>Identification of new drug classification terms in textual resources</article-title>
. <source>Bioinformatics</source>
<volume>23</volume>
: <fpage>i264</fpage>
–<lpage>272</lpage>
.<pub-id pub-id-type="pmid">17646305</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Klinger1"><label>5</label>
<mixed-citation publication-type="journal"><name><surname>Klinger</surname>
<given-names>R</given-names>
</name>
, <name><surname>Kolarik</surname>
<given-names>C</given-names>
</name>
, <name><surname>Fluck</surname>
<given-names>J</given-names>
</name>
, <name><surname>Hofmann-Apitius</surname>
<given-names>M</given-names>
</name>
, <name><surname>Friedrich</surname>
<given-names>CM</given-names>
</name>
(<year>2008</year>
) <article-title>Detection of IUPAC and IUPAC-like chemical names</article-title>
. <source>Bioinformatics</source>
<volume>24</volume>
: <fpage>i268</fpage>
–<lpage>276</lpage>
.<pub-id pub-id-type="pmid">18586724</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Zimmermann1"><label>6</label>
<mixed-citation publication-type="journal"><name><surname>Zimmermann</surname>
<given-names>M</given-names>
</name>
, <name><surname>Fluck</surname>
<given-names>J</given-names>
</name>
, <name><surname>Thi</surname>
<given-names>LT</given-names>
</name>
, <name><surname>Kolarik</surname>
<given-names>C</given-names>
</name>
, <name><surname>Kumpf</surname>
<given-names>K</given-names>
</name>
, <etal>et al</etal>
(<year>2005</year>
) <article-title>Information extraction in the life sciences: perspectives for medicinal chemistry, pharmacology and toxicology</article-title>
. <source>Curr Top Med Chem</source>
<volume>5</volume>
: <fpage>785</fpage>
–<lpage>796</lpage>
.<pub-id pub-id-type="pmid">16101418</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-European1"><label>7</label>
<mixed-citation publication-type="other">European Patent Office. Available: <ext-link ext-link-type="uri" xlink:href="http://www.epo.org/">http://www.epo.org/</ext-link>
. Accessed 2014 Jan 1.</mixed-citation>
</ref>
<ref id="pone.0107477-United1"><label>8</label>
<mixed-citation publication-type="other">United States Patent and Trademark Office. Available: <ext-link ext-link-type="uri" xlink:href="http://www.uspto.gov/">http://www.uspto.gov/</ext-link>
. Accessed 2014 Jan 1.</mixed-citation>
</ref>
<ref id="pone.0107477-World1"><label>9</label>
<mixed-citation publication-type="other">World Intellectual Property Organization. Available: <ext-link ext-link-type="uri" xlink:href="http://www.wipo.int/">http://www.wipo.int/</ext-link>
. Accessed 2014 Jan 1.</mixed-citation>
</ref>
<ref id="pone.0107477-Tseng1"><label>10</label>
<mixed-citation publication-type="journal"><name><surname>Tseng</surname>
<given-names>Y-H</given-names>
</name>
, <name><surname>Lin</surname>
<given-names>C-J</given-names>
</name>
, <name><surname>Lin</surname>
<given-names>Y-I</given-names>
</name>
(<year>2007</year>
) <article-title>Text mining techniques for patent analysis</article-title>
. <source>Inf Process Manag</source>
<volume>43</volume>
: <fpage>1216</fpage>
–<lpage>1247</lpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Jessop1"><label>11</label>
<mixed-citation publication-type="journal"><name><surname>Jessop</surname>
<given-names>DM</given-names>
</name>
, <name><surname>Adams</surname>
<given-names>SE</given-names>
</name>
, <name><surname>Murray-Rust</surname>
<given-names>P</given-names>
</name>
(<year>2011</year>
) <article-title>Mining chemical information from open patents</article-title>
. <source>J Cheminform</source>
<volume>3</volume>
: <fpage>40</fpage>
.<pub-id pub-id-type="pmid">21999425</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Kiss1"><label>12</label>
<mixed-citation publication-type="journal"><name><surname>Kiss</surname>
<given-names>M</given-names>
</name>
, <name><surname>Nagy</surname>
<given-names>Á</given-names>
</name>
, <name><surname>Vincze</surname>
<given-names>V</given-names>
</name>
, <name><surname>Almási</surname>
<given-names>A</given-names>
</name>
, <name><surname>Alexin</surname>
<given-names>Z</given-names>
</name>
, <etal>et al</etal>
(<year>2012</year>
) <article-title>A Manually Annotated Corpus of Pharmaceutical Patents. Text, Speech and Dialogue</article-title>
. <source>Springer Berlin Heidelberg</source>
pp. <fpage>135</fpage>
–<lpage>142</lpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Vazquez1"><label>13</label>
<mixed-citation publication-type="journal"><name><surname>Vazquez</surname>
<given-names>M</given-names>
</name>
, <name><surname>Krallinger</surname>
<given-names>M</given-names>
</name>
, <name><surname>Leitner</surname>
<given-names>F</given-names>
</name>
, <name><surname>Valencia</surname>
<given-names>A</given-names>
</name>
(<year>2011</year>
) <article-title>Text mining for drugs and chemical compounds: methods, tools and applications</article-title>
. <source>Mol Inform</source>
<volume>30</volume>
: <fpage>506</fpage>
–<lpage>519</lpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Corbett1"><label>14</label>
<mixed-citation publication-type="other">Corbett P, Batchelor C, Teufel S (2007) Annotation of chemical named entities. Proceedings of the Workshop on BioNLP 2007 Biological, Translational, and Clinical Language Processing - BioNLP '07. Morristown, NJ, USA: Association for Computational Linguistics. p. 57. Available: <ext-link ext-link-type="uri" xlink:href="http://portal.acm.org/citation.cfm?doid=1572392.1572403">http://portal.acm.org/citation.cfm?doid=1572392.1572403</ext-link>
. Accessed 2014 Jul 10.</mixed-citation>
</ref>
<ref id="pone.0107477-Kim1"><label>15</label>
<mixed-citation publication-type="journal"><name><surname>Kim</surname>
<given-names>JD</given-names>
</name>
, <name><surname>Ohta</surname>
<given-names>T</given-names>
</name>
, <name><surname>Tateisi</surname>
<given-names>Y</given-names>
</name>
, <name><surname>Tsujii</surname>
<given-names>J</given-names>
</name>
(<year>2003</year>
) <article-title>GENIA corpus—semantically annotated corpus for bio-textmining</article-title>
. <source>Bioinformatics</source>
<volume>19</volume>
Suppl 1<fpage>i180</fpage>
–<lpage>182</lpage>
.<pub-id pub-id-type="pmid">12855455</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Kulick1"><label>16</label>
<mixed-citation publication-type="journal"><name><surname>Kulick</surname>
<given-names>S</given-names>
</name>
, <name><surname>Bies</surname>
<given-names>A</given-names>
</name>
, <name><surname>Liberman</surname>
<given-names>M</given-names>
</name>
, <name><surname>Mandel</surname>
<given-names>M</given-names>
</name>
, <name><surname>McDonald</surname>
<given-names>R</given-names>
</name>
, <etal>et al</etal>
(<year>2004</year>
) <article-title>Integrated annotation for biomedical information extraction</article-title>
. <source>Proc. of the Human Language Technology Conference and the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL)</source>
pp. <fpage>61</fpage>
–<lpage>68</lpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Kolrik1"><label>17</label>
<mixed-citation publication-type="journal"><name><surname>Kolárik</surname>
<given-names>C</given-names>
</name>
, <name><surname>Klinger</surname>
<given-names>R</given-names>
</name>
, <name><surname>Friedrich</surname>
<given-names>CM</given-names>
</name>
, <name><surname>Hofmann-Apitius</surname>
<given-names>M</given-names>
</name>
, <name><surname>Fluck</surname>
<given-names>J</given-names>
</name>
(<year>2008</year>
) <article-title>Chemical names: terminological resources and corpora annotation</article-title>
. <source>Workshop on Building and evaluating resources for biomedical text mining</source>
</mixed-citation>
</ref>
<ref id="pone.0107477-Krallinger1"><label>18</label>
<mixed-citation publication-type="journal"><name><surname>Krallinger</surname>
<given-names>M</given-names>
</name>
, <name><surname>Leitner</surname>
<given-names>F</given-names>
</name>
, <name><surname>Rabal</surname>
<given-names>O</given-names>
</name>
, <name><surname>Vazquez</surname>
<given-names>M</given-names>
</name>
, <name><surname>Oyarzabal</surname>
<given-names>J</given-names>
</name>
, <etal>et al</etal>
<article-title>Overview of the chemical compound and drug name recognition (CHEMDNER) task; 2013</article-title>
. <source>BioCreative Challenge Evaluation Workshop</source>
<volume>vol. 2</volume>
pp. <fpage>2</fpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Grego1"><label>19</label>
<mixed-citation publication-type="journal"><name><surname>Grego</surname>
<given-names>T</given-names>
</name>
, <name><surname>Pęzik</surname>
<given-names>P</given-names>
</name>
, <name><surname>Couto</surname>
<given-names>FM</given-names>
</name>
, <name><surname>Rebholz-Schuhmann</surname>
<given-names>D</given-names>
</name>
(<year>2009</year>
) <article-title>Identification of chemical entities in patent documents</article-title>
. <source>Distributed Computing, Artificial Intelligence, Bioinformatics, Soft Computing, and Ambient Assisted Living: Springer</source>
pp. <fpage>942</fpage>
–<lpage>949</lpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Degtyarenko1"><label>20</label>
<mixed-citation publication-type="journal"><name><surname>Degtyarenko</surname>
<given-names>K</given-names>
</name>
, <name><surname>De Matos</surname>
<given-names>P</given-names>
</name>
, <name><surname>Ennis</surname>
<given-names>M</given-names>
</name>
, <name><surname>Hastings</surname>
<given-names>J</given-names>
</name>
, <name><surname>Zbinden</surname>
<given-names>M</given-names>
</name>
, <etal>et al</etal>
(<year>2008</year>
) <article-title>ChEBI: a database and ontology for chemical entities of biological interest</article-title>
. <source>Nucleic Acids Res</source>
<volume>36</volume>
: <fpage>D344</fpage>
–<lpage>D350</lpage>
.<pub-id pub-id-type="pmid">17932057</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Tiago1"><label>21</label>
<mixed-citation publication-type="other">Tiago G, Catia P, Bastos Hugo P (2012) Chemical entity recognition and resolution to ChEBI. ISRN Bioinformatics 2012.</mixed-citation>
</ref>
<ref id="pone.0107477-DeMatos1"><label>22</label>
<mixed-citation publication-type="journal"><name><surname>De Matos</surname>
<given-names>P</given-names>
</name>
, <name><surname>Alcántara</surname>
<given-names>R</given-names>
</name>
, <name><surname>Dekker</surname>
<given-names>A</given-names>
</name>
, <name><surname>Ennis</surname>
<given-names>M</given-names>
</name>
, <name><surname>Hastings</surname>
<given-names>J</given-names>
</name>
, <etal>et al</etal>
(<year>2010</year>
) <article-title>Chemical entities of biological interest: an update</article-title>
. <source>Nucleic Acids Res</source>
<volume>38</volume>
: <fpage>D249</fpage>
–<lpage>D254</lpage>
.<pub-id pub-id-type="pmid">19854951</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-GVK1"><label>23</label>
<mixed-citation publication-type="other">GVK BIO Target Class Based Compound Database. Available: <ext-link ext-link-type="uri" xlink:href="http://www.gvkbio.com/products-services/informatics-analytics/products/standalone-databases/">http://www.gvkbio.com/products-services/informatics-analytics/products/standalone-databases/</ext-link>
. Accessed 2014 Jan 1.</mixed-citation>
</ref>
<ref id="pone.0107477-Sayle1"><label>24</label>
<mixed-citation publication-type="journal"><name><surname>Sayle</surname>
<given-names>R</given-names>
</name>
, <name><surname>Xie</surname>
<given-names>PH</given-names>
</name>
, <name><surname>Muresan</surname>
<given-names>S</given-names>
</name>
(<year>2012</year>
) <article-title>Improved chemical text mining of patents with infinite dictionaries and automatic spelling correction</article-title>
. <source>J Chem Inf Model</source>
<volume>52</volume>
: <fpage>51</fpage>
–<lpage>62</lpage>
.<pub-id pub-id-type="pmid">22148717</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-Akhondi1"><label>25</label>
<mixed-citation publication-type="journal"><name><surname>Akhondi</surname>
<given-names>SA</given-names>
</name>
, <name><surname>Kors</surname>
<given-names>JA</given-names>
</name>
, <name><surname>Muresan</surname>
<given-names>S</given-names>
</name>
(<year>2012</year>
) <article-title>Consistency of systematic chemical identifiers within and between small-molecule databases</article-title>
. <source>J Cheminf</source>
<volume>4</volume>
: <fpage>35</fpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-About1"><label>26</label>
<mixed-citation publication-type="other">About IUPAC. Available: <ext-link ext-link-type="uri" xlink:href="http://www.iupac.org/home/about.html/">http://www.iupac.org/home/about.html/</ext-link>
. Accessed 2014 Jan 1.</mixed-citation>
</ref>
<ref id="pone.0107477-Weininger1"><label>27</label>
<mixed-citation publication-type="journal"><name><surname>Weininger</surname>
<given-names>D</given-names>
</name>
(<year>1988</year>
) <article-title>SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules</article-title>
. <source>J Chem Inf Comput Sci</source>
<volume>28</volume>
: <fpage>31</fpage>
–<lpage>36</lpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Heller1"><label>28</label>
<mixed-citation publication-type="journal"><name><surname>Heller</surname>
<given-names>S</given-names>
</name>
, <name><surname>McNaught</surname>
<given-names>A</given-names>
</name>
, <name><surname>Stein</surname>
<given-names>S</given-names>
</name>
, <name><surname>Tchekhovskoi</surname>
<given-names>D</given-names>
</name>
, <name><surname>Pletnev</surname>
<given-names>I</given-names>
</name>
(<year>2013</year>
) <article-title>InChI - the worldwide chemical structure identifier standard</article-title>
. <source>J Cheminform</source>
<volume>5</volume>
: <fpage>7</fpage>
.<pub-id pub-id-type="pmid">23343401</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0107477-About2"><label>29</label>
<mixed-citation publication-type="other">About the InChI Standard. Available: <ext-link ext-link-type="uri" xlink:href="http://www.inchi-trust.org/about-the-inchi-standard/">http://www.inchi-trust.org/about-the-inchi-standard/</ext-link>
. Accessed 2014 Jan 1.</mixed-citation>
</ref>
<ref id="pone.0107477-Undefined1"><label>30</label>
<mixed-citation publication-type="journal"><collab>Undefined</collab>
(<year>1978</year>
) <article-title>CAS Registry System</article-title>
. <source>J Chem Inf Model</source>
<volume>18</volume>
: <fpage>58</fpage>
–<lpage>58</lpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-CAS1"><label>31</label>
<mixed-citation publication-type="other">CAS REGISTRY - The gold standard for chemical substance information. Available: <ext-link ext-link-type="uri" xlink:href="http://www.cas.org/content/chemical-substances/">http://www.cas.org/content/chemical-substances/</ext-link>
. Accessed 2014 Jan 1.</mixed-citation>
</ref>
<ref id="pone.0107477-Krushinski1"><label>32</label>
<mixed-citation publication-type="other">Krushinski JH, Robertson DW, Wong DT (1991) 3-aryloxy-3-substituted propanamines. USPTO US5023269 A.</mixed-citation>
</ref>
<ref id="pone.0107477-Villani1"><label>33</label>
<mixed-citation publication-type="other">Villani FJ, Wong JK (1987) Antihistaminic 8-(halo)-substituted 6,11-dihydro-11-(4-piperidylidene)-5H-benzo[5,6]cyclohepta[1,2-b]pyridines. USPTO US4659716 A.</mixed-citation>
</ref>
<ref id="pone.0107477-Lowe1"><label>34</label>
<mixed-citation publication-type="journal"><name><surname>Lowe</surname>
<given-names>DM</given-names>
</name>
, <name><surname>Sayle</surname>
<given-names>RA</given-names>
</name>
(<year>2013</year>
) <article-title>LeadMine: A grammar and dictionary driven approach to chemical entity recognition</article-title>
. <source>BioCreative Challenge Evaluation Workshop</source>
<volume>vol. 2</volume>
pp. <fpage>47</fpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Stenetorp1"><label>35</label>
<mixed-citation publication-type="journal"><name><surname>Stenetorp</surname>
<given-names>P</given-names>
</name>
, <name><surname>Pyysalo</surname>
<given-names>S</given-names>
</name>
, <name><surname>Topić</surname>
<given-names>G</given-names>
</name>
, <name><surname>Ohta</surname>
<given-names>T</given-names>
</name>
, <name><surname>Ananiadou</surname>
<given-names>S</given-names>
</name>
, <etal>et al</etal>
(<year>2012</year>
) <article-title>BRAT: a web-based tool for NLP-assisted text annotation</article-title>
. <source>Association for Computational Linguistics</source>
pp. <fpage>102</fpage>
–<lpage>107</lpage>
.</mixed-citation>
</ref>
<ref id="pone.0107477-Lewin1"><label>36</label>
<mixed-citation publication-type="journal"><name><surname>Lewin</surname>
<given-names>I</given-names>
</name>
, <name><surname>Kafkas</surname>
<given-names>S</given-names>
</name>
, <name><surname>Rebholz-Schuhmann</surname>
<given-names>D</given-names>
</name>
(<year>2012</year>
) <article-title>Centroids: Gold standards with distributional variation</article-title>
. <source>LREC</source>
pp. <fpage>3894</fpage>
–<lpage>3900</lpage>
.</mixed-citation>
</ref>
</ref-list>
</back>
</pmc>
<affiliations><list><country><li>Allemagne</li>
<li>Inde</li>
<li>Pays-Bas</li>
<li>Royaume-Uni</li>
<li>Suède</li>
</country>
<region><li>Angleterre</li>
<li>District de Cologne</li>
<li>Hollande-Méridionale</li>
<li>Rhénanie-du-Nord-Westphalie</li>
</region>
<settlement><li>Rotterdam</li>
<li>Sankt Augustin</li>
</settlement>
</list>
<tree><country name="Pays-Bas"><region name="Hollande-Méridionale"><name sortKey="Akhondi, Saber A" sort="Akhondi, Saber A" uniqKey="Akhondi S" first="Saber A." last="Akhondi">Saber A. Akhondi</name>
</region>
<name sortKey="Kors, Jan A" sort="Kors, Jan A" uniqKey="Kors J" first="Jan A." last="Kors">Jan A. Kors</name>
</country>
<country name="Allemagne"><region name="Rhénanie-du-Nord-Westphalie"><name sortKey="Klenner, Alexander G" sort="Klenner, Alexander G" uniqKey="Klenner A" first="Alexander G." last="Klenner">Alexander G. Klenner</name>
</region>
<name sortKey="Zimmermann, Marc" sort="Zimmermann, Marc" uniqKey="Zimmermann M" first="Marc" last="Zimmermann">Marc Zimmermann</name>
</country>
<country name="Suède"><noRegion><name sortKey="Tyrchan, Christian" sort="Tyrchan, Christian" uniqKey="Tyrchan C" first="Christian" last="Tyrchan">Christian Tyrchan</name>
</noRegion>
<name sortKey="Muresan, Sorel" sort="Muresan, Sorel" uniqKey="Muresan S" first="Sorel" last="Muresan">Sorel Muresan</name>
</country>
<country name="Inde"><noRegion><name sortKey="Manchala, Anil K" sort="Manchala, Anil K" uniqKey="Manchala A" first="Anil K." last="Manchala">Anil K. Manchala</name>
</noRegion>
<name sortKey="Boppana, Kiran" sort="Boppana, Kiran" uniqKey="Boppana K" first="Kiran" last="Boppana">Kiran Boppana</name>
<name sortKey="Jagarlapudi, Sarma A R P" sort="Jagarlapudi, Sarma A R P" uniqKey="Jagarlapudi S" first="Sarma A. R. P." last="Jagarlapudi">Sarma A. R. P. Jagarlapudi</name>
</country>
<country name="Royaume-Uni"><region name="Angleterre"><name sortKey="Lowe, Daniel" sort="Lowe, Daniel" uniqKey="Lowe D" first="Daniel" last="Lowe">Daniel Lowe</name>
</region>
<name sortKey="Sayle, Roger" sort="Sayle, Roger" uniqKey="Sayle R" first="Roger" last="Sayle">Roger Sayle</name>
</country>
</tree>
</affiliations>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/OcrV1/Data/Pmc/Checkpoint
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000057 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Checkpoint/biblio.hfd -nk 000057 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Ticri/CIDE |area= OcrV1 |flux= Pmc |étape= Checkpoint |type= RBID |clé= PMC:4182036 |texte= Annotated Chemical Patent Corpus: A Gold Standard for Text Mining }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Checkpoint/RBID.i -Sk "pubmed:25268232" \ | HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Checkpoint/biblio.hfd \ | NlmPubMed2Wicri -a OcrV1
![]() | This area was generated with Dilib version V0.6.32. | ![]() |