Redundancy-Aware Topic Modeling for Patient Record Notes
Identifieur interne : 000042 ( Pmc/Checkpoint ); précédent : 000041; suivant : 000043Redundancy-Aware Topic Modeling for Patient Record Notes
Auteurs : Raphael Cohen [Israël] ; Iddo Aviram [Israël] ; Michael Elhadad [Israël] ; Noémie Elhadad [États-Unis]Source :
- PLoS ONE [ 1932-6203 ] ; 2014.
Abstract
The clinical notes in a given patient record contain much redundancy, in large part due to clinicians’ documentation habit of copying from previous notes in the record and pasting into a new note. Previous work has shown that this redundancy has a negative impact on the quality of text mining and topic modeling in particular. In this paper we describe a novel variant of Latent Dirichlet Allocation (LDA) topic modeling, Red-LDA, which takes into account the inherent redundancy of patient records when modeling content of clinical notes. To assess the value of Red-LDA, we experiment with three baselines and our novel redundancy-aware topic modeling method: given a large collection of patient records, (i) apply vanilla LDA to all documents in all input records; (ii) identify and remove all redundancy by chosing a single representative document for each record as input to LDA; (iii) identify and remove all redundant paragraphs in each record, leaving partial, non-redundant documents as input to LDA; and (iv) apply Red-LDA to all documents in all input records. Both quantitative evaluation carried out through log-likelihood on held-out data and topic coherence of produced topics and qualitative assessement of topics carried out by physicians show that Red-LDA produces superior models to all three baseline strategies. This research contributes to the emerging field of understanding the characteristics of the electronic health record and how to account for them in the framework of data mining. The code for the two redundancy-elimination baselines and Red-LDA is made publicly available to the community.
Url:
DOI: 10.1371/journal.pone.0087555
PubMed: 24551060
PubMed Central: 3923754
Affiliations:
Links toward previous steps (curation, corpus...)
Links to Exploration step
PMC:3923754Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Redundancy-Aware Topic Modeling for Patient Record Notes</title>
<author><name sortKey="Cohen, Raphael" sort="Cohen, Raphael" uniqKey="Cohen R" first="Raphael" last="Cohen">Raphael Cohen</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><addr-line>Department of Computer Science, Ben Gurion University, Beer Sheva, Israel</addr-line>
</nlm:aff>
<country xml:lang="fr">Israël</country>
<wicri:regionArea>Department of Computer Science, Ben Gurion University, Beer Sheva</wicri:regionArea>
<wicri:noRegion>Beer Sheva</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Aviram, Iddo" sort="Aviram, Iddo" uniqKey="Aviram I" first="Iddo" last="Aviram">Iddo Aviram</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><addr-line>Department of Computer Science, Ben Gurion University, Beer Sheva, Israel</addr-line>
</nlm:aff>
<country xml:lang="fr">Israël</country>
<wicri:regionArea>Department of Computer Science, Ben Gurion University, Beer Sheva</wicri:regionArea>
<wicri:noRegion>Beer Sheva</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Elhadad, Michael" sort="Elhadad, Michael" uniqKey="Elhadad M" first="Michael" last="Elhadad">Michael Elhadad</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><addr-line>Department of Computer Science, Ben Gurion University, Beer Sheva, Israel</addr-line>
</nlm:aff>
<country xml:lang="fr">Israël</country>
<wicri:regionArea>Department of Computer Science, Ben Gurion University, Beer Sheva</wicri:regionArea>
<wicri:noRegion>Beer Sheva</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Elhadad, Noemie" sort="Elhadad, Noemie" uniqKey="Elhadad N" first="Noémie" last="Elhadad">Noémie Elhadad</name>
<affiliation wicri:level="4"><nlm:aff id="aff2"><addr-line>Department of Biomedical Informatics, Columbia University, New York, New York, United States of America</addr-line>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>Department of Biomedical Informatics, Columbia University, New York, New York</wicri:regionArea>
<placeName><region type="state">État de New York</region>
<settlement type="city">New York</settlement>
</placeName>
<orgName type="university">Université Columbia</orgName>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PMC</idno>
<idno type="pmid">24551060</idno>
<idno type="pmc">3923754</idno>
<idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3923754</idno>
<idno type="RBID">PMC:3923754</idno>
<idno type="doi">10.1371/journal.pone.0087555</idno>
<date when="2014">2014</date>
<idno type="wicri:Area/Pmc/Corpus">000177</idno>
<idno type="wicri:Area/Pmc/Curation">000177</idno>
<idno type="wicri:Area/Pmc/Checkpoint">000042</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Redundancy-Aware Topic Modeling for Patient Record Notes</title>
<author><name sortKey="Cohen, Raphael" sort="Cohen, Raphael" uniqKey="Cohen R" first="Raphael" last="Cohen">Raphael Cohen</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><addr-line>Department of Computer Science, Ben Gurion University, Beer Sheva, Israel</addr-line>
</nlm:aff>
<country xml:lang="fr">Israël</country>
<wicri:regionArea>Department of Computer Science, Ben Gurion University, Beer Sheva</wicri:regionArea>
<wicri:noRegion>Beer Sheva</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Aviram, Iddo" sort="Aviram, Iddo" uniqKey="Aviram I" first="Iddo" last="Aviram">Iddo Aviram</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><addr-line>Department of Computer Science, Ben Gurion University, Beer Sheva, Israel</addr-line>
</nlm:aff>
<country xml:lang="fr">Israël</country>
<wicri:regionArea>Department of Computer Science, Ben Gurion University, Beer Sheva</wicri:regionArea>
<wicri:noRegion>Beer Sheva</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Elhadad, Michael" sort="Elhadad, Michael" uniqKey="Elhadad M" first="Michael" last="Elhadad">Michael Elhadad</name>
<affiliation wicri:level="1"><nlm:aff id="aff1"><addr-line>Department of Computer Science, Ben Gurion University, Beer Sheva, Israel</addr-line>
</nlm:aff>
<country xml:lang="fr">Israël</country>
<wicri:regionArea>Department of Computer Science, Ben Gurion University, Beer Sheva</wicri:regionArea>
<wicri:noRegion>Beer Sheva</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Elhadad, Noemie" sort="Elhadad, Noemie" uniqKey="Elhadad N" first="Noémie" last="Elhadad">Noémie Elhadad</name>
<affiliation wicri:level="4"><nlm:aff id="aff2"><addr-line>Department of Biomedical Informatics, Columbia University, New York, New York, United States of America</addr-line>
</nlm:aff>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>Department of Biomedical Informatics, Columbia University, New York, New York</wicri:regionArea>
<placeName><region type="state">État de New York</region>
<settlement type="city">New York</settlement>
</placeName>
<orgName type="university">Université Columbia</orgName>
</affiliation>
</author>
</analytic>
<series><title level="j">PLoS ONE</title>
<idno type="eISSN">1932-6203</idno>
<imprint><date when="2014">2014</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en"><p>The clinical notes in a given patient record contain much redundancy, in large part due to clinicians’ documentation habit of copying from previous notes in the record and pasting into a new note. Previous work has shown that this redundancy has a negative impact on the quality of text mining and topic modeling in particular. In this paper we describe a novel variant of Latent Dirichlet Allocation (LDA) topic modeling, Red-LDA, which takes into account the inherent redundancy of patient records when modeling content of clinical notes. To assess the value of Red-LDA, we experiment with three baselines and our novel redundancy-aware topic modeling method: given a large collection of patient records, (i) apply vanilla LDA to all documents in all input records; (ii) identify and remove all redundancy by chosing a single representative document for each record as input to LDA; (iii) identify and remove all redundant paragraphs in each record, leaving partial, non-redundant documents as input to LDA; and (iv) apply Red-LDA to all documents in all input records. Both quantitative evaluation carried out through log-likelihood on held-out data and topic coherence of produced topics and qualitative assessement of topics carried out by physicians show that Red-LDA produces superior models to all three baseline strategies. This research contributes to the emerging field of understanding the characteristics of the electronic health record and how to account for them in the framework of data mining. The code for the two redundancy-elimination baselines and Red-LDA is made publicly available to the community.</p>
</div>
</front>
<back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Siegler, E" uniqKey="Siegler E">E Siegler</name>
</author>
<author><name sortKey="Adelman, R" uniqKey="Adelman R">R Adelman</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Markel, A" uniqKey="Markel A">A Markel</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Cohen, R" uniqKey="Cohen R">R Cohen</name>
</author>
<author><name sortKey="Elhadad, M" uniqKey="Elhadad M">M Elhadad</name>
</author>
<author><name sortKey="Elhadad, N" uniqKey="Elhadad N">N Elhadad</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Blei, D" uniqKey="Blei D">D Blei</name>
</author>
<author><name sortKey="Ng, A" uniqKey="Ng A">A Ng</name>
</author>
<author><name sortKey="Jordan, M" uniqKey="Jordan M">M Jordan</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Wang, H" uniqKey="Wang H">H Wang</name>
</author>
<author><name sortKey="Ding, Y" uniqKey="Ding Y">Y Ding</name>
</author>
<author><name sortKey="Tang, J" uniqKey="Tang J">J Tang</name>
</author>
<author><name sortKey="Dong, X" uniqKey="Dong X">X Dong</name>
</author>
<author><name sortKey="He, B" uniqKey="He B">B He</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Wallach, H" uniqKey="Wallach H">H Wallach</name>
</author>
<author><name sortKey="Murray, I" uniqKey="Murray I">I Murray</name>
</author>
<author><name sortKey="Salakhutdinov, R" uniqKey="Salakhutdinov R">R Salakhutdinov</name>
</author>
<author><name sortKey="Mimno, D" uniqKey="Mimno D">D Mimno</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Bisgin, H" uniqKey="Bisgin H">H Bisgin</name>
</author>
<author><name sortKey="Liu, Z" uniqKey="Liu Z">Z Liu</name>
</author>
<author><name sortKey="Fang, H" uniqKey="Fang H">H Fang</name>
</author>
<author><name sortKey="Xu, X" uniqKey="Xu X">X Xu</name>
</author>
<author><name sortKey="Tong, W" uniqKey="Tong W">W Tong</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Lippincott, T" uniqKey="Lippincott T">T Lippincott</name>
</author>
<author><name sortKey="Seaghdha, D" uniqKey="Seaghdha D">DÓ Séaghdha</name>
</author>
<author><name sortKey="Korhonen, A" uniqKey="Korhonen A">A Korhonen</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Bisgin, H" uniqKey="Bisgin H">H Bisgin</name>
</author>
<author><name sortKey="Liu, Z" uniqKey="Liu Z">Z Liu</name>
</author>
<author><name sortKey="Kelly, R" uniqKey="Kelly R">R Kelly</name>
</author>
<author><name sortKey="Fang, H" uniqKey="Fang H">H Fang</name>
</author>
<author><name sortKey="Xu, X" uniqKey="Xu X">X Xu</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Holm, L" uniqKey="Holm L">L Holm</name>
</author>
<author><name sortKey="Sander, C" uniqKey="Sander C">C Sander</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Buslje, C" uniqKey="Buslje C">C Buslje</name>
</author>
<author><name sortKey="Santos, J" uniqKey="Santos J">J Santos</name>
</author>
<author><name sortKey="Delfino, J" uniqKey="Delfino J">J Delfino</name>
</author>
<author><name sortKey="Nielsen, M" uniqKey="Nielsen M">M Nielsen</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Hripcsak, G" uniqKey="Hripcsak G">G Hripcsak</name>
</author>
<author><name sortKey="Albers, D" uniqKey="Albers D">D Albers</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
</listBibl>
</div1>
</back>
</TEI>
<pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">PLoS One</journal-id>
<journal-id journal-id-type="iso-abbrev">PLoS ONE</journal-id>
<journal-id journal-id-type="publisher-id">plos</journal-id>
<journal-id journal-id-type="pmc">plosone</journal-id>
<journal-title-group><journal-title>PLoS ONE</journal-title>
</journal-title-group>
<issn pub-type="epub">1932-6203</issn>
<publisher><publisher-name>Public Library of Science</publisher-name>
<publisher-loc>San Francisco, USA</publisher-loc>
</publisher>
</journal-meta>
<article-meta><article-id pub-id-type="pmid">24551060</article-id>
<article-id pub-id-type="pmc">3923754</article-id>
<article-id pub-id-type="publisher-id">PONE-D-13-38704</article-id>
<article-id pub-id-type="doi">10.1371/journal.pone.0087555</article-id>
<article-categories><subj-group subj-group-type="heading"><subject>Research Article</subject>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Computer science</subject>
<subj-group><subject>Algorithms</subject>
</subj-group>
<subj-group><subject>Computer modeling</subject>
</subj-group>
<subj-group><subject>Computing methods</subject>
<subj-group><subject>Computer inferencing</subject>
</subj-group>
</subj-group>
<subj-group><subject>Natural language processing</subject>
</subj-group>
<subj-group><subject>Text mining</subject>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Mathematics</subject>
<subj-group><subject>Applied mathematics</subject>
<subj-group><subject>Algorithms</subject>
<subject>Complex systems</subject>
</subj-group>
</subj-group>
<subj-group><subject>Mathematical computing</subject>
</subj-group>
</subj-group>
<subj-group subj-group-type="Discipline-v2"><subject>Medicine</subject>
<subj-group><subject>Non-clinical medicine</subject>
<subj-group><subject>Health informatics</subject>
</subj-group>
</subj-group>
</subj-group>
</article-categories>
<title-group><article-title>Redundancy-Aware Topic Modeling for Patient Record Notes</article-title>
<alt-title alt-title-type="running-head">Redundancy-Aware Topic Modeling for Patient Notes</alt-title>
</title-group>
<contrib-group><contrib contrib-type="author"><name><surname>Cohen</surname>
<given-names>Raphael</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
<xref ref-type="corresp" rid="cor1"><sup>*</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Aviram</surname>
<given-names>Iddo</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Elhadad</surname>
<given-names>Michael</given-names>
</name>
<xref ref-type="aff" rid="aff1"><sup>1</sup>
</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Elhadad</surname>
<given-names>Noémie</given-names>
</name>
<xref ref-type="aff" rid="aff2"><sup>2</sup>
</xref>
</contrib>
</contrib-group>
<aff id="aff1"><label>1</label>
<addr-line>Department of Computer Science, Ben Gurion University, Beer Sheva, Israel</addr-line>
</aff>
<aff id="aff2"><label>2</label>
<addr-line>Department of Biomedical Informatics, Columbia University, New York, New York, United States of America</addr-line>
</aff>
<contrib-group><contrib contrib-type="editor"><name><surname>Khanin</surname>
<given-names>Raya</given-names>
</name>
<role>Editor</role>
<xref ref-type="aff" rid="edit1"></xref>
</contrib>
</contrib-group>
<aff id="edit1"><addr-line>Memorial Sloan Kettering Cancer Center, United States of America</addr-line>
</aff>
<author-notes><corresp id="cor1">* E-mail: <email>cohenrap@cs.bgu.ac.il</email>
</corresp>
<fn fn-type="conflict"><p><bold>Competing Interests: </bold>
The authors have declared that no competing interests exist.</p>
</fn>
<fn fn-type="con"><p>Conceived and designed the experiments: RC NE ME. Performed the experiments: RC IA. Analyzed the data: RC. Wrote the paper: RC ME NE.</p>
</fn>
</author-notes>
<pub-date pub-type="collection"><year>2014</year>
</pub-date>
<pub-date pub-type="epub"><day>13</day>
<month>2</month>
<year>2014</year>
</pub-date>
<volume>9</volume>
<issue>2</issue>
<elocation-id>e87555</elocation-id>
<history><date date-type="received"><day>21</day>
<month>9</month>
<year>2013</year>
</date>
<date date-type="accepted"><day>23</day>
<month>12</month>
<year>2013</year>
</date>
</history>
<permissions><copyright-year>2014</copyright-year>
<copyright-holder>Cohen et al</copyright-holder>
<license><license-p>This is an open-access article distributed under the terms of the <ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution License</ext-link>
, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.</license-p>
</license>
</permissions>
<abstract><p>The clinical notes in a given patient record contain much redundancy, in large part due to clinicians’ documentation habit of copying from previous notes in the record and pasting into a new note. Previous work has shown that this redundancy has a negative impact on the quality of text mining and topic modeling in particular. In this paper we describe a novel variant of Latent Dirichlet Allocation (LDA) topic modeling, Red-LDA, which takes into account the inherent redundancy of patient records when modeling content of clinical notes. To assess the value of Red-LDA, we experiment with three baselines and our novel redundancy-aware topic modeling method: given a large collection of patient records, (i) apply vanilla LDA to all documents in all input records; (ii) identify and remove all redundancy by chosing a single representative document for each record as input to LDA; (iii) identify and remove all redundant paragraphs in each record, leaving partial, non-redundant documents as input to LDA; and (iv) apply Red-LDA to all documents in all input records. Both quantitative evaluation carried out through log-likelihood on held-out data and topic coherence of produced topics and qualitative assessement of topics carried out by physicians show that Red-LDA produces superior models to all three baseline strategies. This research contributes to the emerging field of understanding the characteristics of the electronic health record and how to account for them in the framework of data mining. The code for the two redundancy-elimination baselines and Red-LDA is made publicly available to the community.</p>
</abstract>
<funding-group><funding-statement>This work was supported by a National Library of Medicine grant R01 LM010027 (NE). Any opinions, findings, or conclusions are those of the authors, and do not necessarily reflect the views of the funding organization. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</funding-statement>
</funding-group>
<counts><page-count count="7"></page-count>
</counts>
</article-meta>
</front>
<body><sec id="s1"><title>Introduction</title>
<p>The information contained in the electronic health record for a given patient record is quite redundant. In the clinical community there is some discontent with copying and pasting <xref rid="pone.0087555-Siegler1" ref-type="bibr">[1]</xref>
, <xref rid="pone.0087555-Markel1" ref-type="bibr">[2]</xref>
and its negative impact on quality of clinical documentation. In previous work, we have shown through a quantitative analysis that redundancy also hurts standard text-mining tools, such as collocation identification and topic modeling <xref rid="pone.0087555-Cohen1" ref-type="bibr">[3]</xref>
. In this paper, we focus on topic modeling for clinical notes.</p>
<p>Topic Modeling with Latent Dirichlet Allocation (LDA) <xref rid="pone.0087555-Blei1" ref-type="bibr">[4]</xref>
is a popular unsupervised method for discovering latent semantic properties of a document collection. Topic modeling has been shown to help in large number of tasks, including document classification and clustering, multi-document summarization <xref rid="pone.0087555-Haghighi1" ref-type="bibr">[5]</xref>
, search <xref rid="pone.0087555-Carterette1" ref-type="bibr">[6]</xref>
, document labeling <xref rid="pone.0087555-Blei2" ref-type="bibr">[7]</xref>
, <xref rid="pone.0087555-Ramage1" ref-type="bibr">[8]</xref>
, and information extraction <xref rid="pone.0087555-Wang1" ref-type="bibr">[9]</xref>
. LDA is known to be sensitive to noise: for example, it is standard practice to remove stop words before applying LDA, and Walker et al. showed that LDA underperforms on documents noisy with optical character recognition (OCR) errors <xref rid="pone.0087555-Walker1" ref-type="bibr">[10]</xref>
. The measure of LDAs sensitivity to different kinds of noise is not well understood, especially as various methods are used for evaluating the produced topic models <xref rid="pone.0087555-Wallach1" ref-type="bibr">[11]</xref>
, <xref rid="pone.0087555-Mimno1" ref-type="bibr">[12]</xref>
.</p>
<p>In the biomedical domain, probabilistic graphical models like LDA are an emerging technology only recently applied to EHR data <xref rid="pone.0087555-Arnold1" ref-type="bibr">[13]</xref>
–<xref rid="pone.0087555-Lehman1" ref-type="bibr">[17]</xref>
, biomedical corpora <xref rid="pone.0087555-Bisgin1" ref-type="bibr">[18]</xref>
–<xref rid="pone.0087555-Bisgin2" ref-type="bibr">[20]</xref>
, and consumer health resources <xref rid="pone.0087555-Crain1" ref-type="bibr">[21]</xref>
. While these initial results are promising on EHR data, initial research suggests that the specific characteristics of the clinical language affect the methods and results of these techniques <xref rid="pone.0087555-Cohen1" ref-type="bibr">[3]</xref>
, <xref rid="pone.0087555-Arnold2" ref-type="bibr">[22]</xref>
, <xref rid="pone.0087555-Halpern1" ref-type="bibr">[23]</xref>
.</p>
<p>In this paper, we describe a novel variant of LDA topic modeling, Red-LDA, which takes into account the inherent redundancy of clinical notes within a given patient record, and produces better topic models, as shown through quantitative and qualitative evaluation. We experiment with three baselines and a novel variant of LDA to handle redundancy: (i) apply vanilla LDA to all documents in all input records; (ii) identify and remove all redundancy by chosing a single representative document for each record as input to LDA; (iii) identify and remove all redundant paragraphs in each record, leaving partial, non-redundant documents as input to LDA; and (iv) apply Red-LDA to all documents in all input records.</p>
<sec id="s1a"><title>Problem Analysis</title>
<p>Characterizing the level of information redundancy between two sentences or two documents can be complex, and in fact has been a topic of study in paraphrasing, information fusion, and textual entailment. In this paper, redundancy is defined through a string similarity metric, very much the same way two sequences in bioinformatics are considered to have a high alignment score. Corpora of patient notes, which contain much copy-and-paste within the notes of a given patient, or document collections of news stories about the same events contain large amount of such textual redundancy. As such simple string similarity metrics are enough to characterize their levels of redundancy.</p>
<p>While in many natural language processing applications, redundancy is considered useful and its presence in a document collection is leveraged <xref rid="pone.0087555-Zanzotto1" ref-type="bibr">[24]</xref>
, in bioinformatics, for instance, it is standard practice to remove redundant sequences from databases <xref rid="pone.0087555-Holm1" ref-type="bibr">[25]</xref>
or to adjust for such sequences when using Gibbs sampling <xref rid="pone.0087555-Buslje1" ref-type="bibr">[26]</xref>
. In Cohen et al. <xref rid="pone.0087555-Cohen1" ref-type="bibr">[3]</xref>
experiments indicate that patient record redundancy negatively affects data-driven methods, and topic modeling in particular.</p>
<p>Let us try to understand where the LDA model assumptions are invalidated when applied to redundant data. The LDA generative model assumes that a document is produced by (i) sampling a Dirichlet distribution for topic mixture in the document, and then (ii) producing each word by choosing a topic from the document-level topic distribution and sampling the selected topic to produce the word. Copy-paste redundancy means that some of the words in a document are not sampled from the documents topics but instead are copied from another document. For example, consider two documents <inline-formula><inline-graphic xlink:href="pone.0087555.e001.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0087555.e002.jpg"></inline-graphic>
</inline-formula>
, where <inline-formula><inline-graphic xlink:href="pone.0087555.e003.jpg"></inline-graphic>
</inline-formula>
was created by the basic LDA generative process, while <inline-formula><inline-graphic xlink:href="pone.0087555.e004.jpg"></inline-graphic>
</inline-formula>
was created by copying <inline-formula><inline-graphic xlink:href="pone.0087555.e005.jpg"></inline-graphic>
</inline-formula>
from <inline-formula><inline-graphic xlink:href="pone.0087555.e006.jpg"></inline-graphic>
</inline-formula>
and then sampling <inline-formula><inline-graphic xlink:href="pone.0087555.e007.jpg"></inline-graphic>
</inline-formula>
using the basic generative process. We would expect the copied words to retain their prior topic assignment (that is, the occurrences of <inline-formula><inline-graphic xlink:href="pone.0087555.e008.jpg"></inline-graphic>
</inline-formula>
in <inline-formula><inline-graphic xlink:href="pone.0087555.e009.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0087555.e010.jpg"></inline-graphic>
</inline-formula>
should share the same topic assignment). In practice, however, the LDA sampling algorithm may assign them to different topics. Even when the copied words are assigned to the same topic, the weight of copied words in their topic is abusively increased. If a word of low probability in topic <inline-formula><inline-graphic xlink:href="pone.0087555.e011.jpg"></inline-graphic>
</inline-formula>
is sampled from <inline-formula><inline-graphic xlink:href="pone.0087555.e012.jpg"></inline-graphic>
</inline-formula>
once and duplicated through copy-paste into many documents, our estimate of topic <inline-formula><inline-graphic xlink:href="pone.0087555.e013.jpg"></inline-graphic>
</inline-formula>
would be misled to give that word higher probability within <inline-formula><inline-graphic xlink:href="pone.0087555.e014.jpg"></inline-graphic>
</inline-formula>
. Such an approach would lead to over-fitting the model to observed data, assuming the probability to copy segments is not directly dependent on their topical relevance.</p>
<p>The main contribution of this paper is a variant of LDA, Red-LDA, which takes into account the fact that words are copied from a source document in the patient records. The next section shows the results of our experiments comparing Red-LDA to three topic-modeling baselines.</p>
</sec>
</sec>
<sec id="s2"><title>Results</title>
<p>To assess the value of handling redundancy explicitly as part of the topic modeling task of clinical notes, we conducted a comparison of the redundancy-aware LDA (Red-LDA) to alternative methods according to two quantitative established metrics for evaluation of topic modeling – log-likelihood and topic coherence – and a qualitative review of generated topics by clinical experts.</p>
<sec id="s2a"><title>Quantitative Analysis</title>
<p>Topic models were learned using four methods on the same training set: vanilla LDA, DeleteDoc-LDA, DeleteWord-LDA, and Red-LDA.</p>
<sec id="s2a1"><title>Log-likelihood</title>
<p>The models produced on the training set were used to produce topic assignments for the held-out documents, and log-likelihood fit of the different models were calculated. All the experiments were repeated three times and the average likelihood is reported (we report standard deviation over 3 runs). <xref ref-type="fig" rid="pone-0087555-g001">Figure 1</xref>
shows the results. The worse performance is obtained for Vanilla LDA trained on the original, redundant corpus. DeleteWord-LDA, which discarded redundant words in the input documents, produces slightly better models. In the DeleteDoc-LDA version, the resulting, non-redundant corpus contained only 40% of all documents in the original corpus. Yet, the topic it generates are superior to the ones generated by the vanilla LDA trained on the redundant, original corpus. DeleteDoc-LDA also outperforms DeleteWord-LDA, which confirms the hypothesis that discarding parts of documents rather than whole documents might hurt overall coherence of a corpus. Finally, Red-LDA outperforms all baseline methods.</p>
<fig id="pone-0087555-g001" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0087555.g001</object-id>
<label>Figure 1</label>
<caption><title>Comparison of four models according to log-likelihood of held-out data.</title>
<p>Red-LDA performs the best, well outside the deviations of the three other models. Training vanilla LDA on just 40% of the documents (DeleteDoc-LDA) produced a better model than training on the entire redundant corpus (Vanilla LDA).</p>
</caption>
<graphic xlink:href="pone.0087555.g001"></graphic>
</fig>
</sec>
<sec id="s2a2"><title>Topic coherence</title>
<p>We use the topic coherence score for evaluating the quality of topics produced by the Red-LDA and Vanilla LDA methods <xref rid="pone.0087555-Mimno1" ref-type="bibr">[12]</xref>
, state-of-the-art method for evaluation (see <xref rid="pone.0087555-Wallach1" ref-type="bibr">[11]</xref>
for earlier work). This measure is based on the co-document frequency of the pairs of the most words probable words in each topic. Thus, the higher the topic coherence metric for a set of topics, the higher quality the topics from a semantic standpoint. All the experiments were repeated three times and the average coherence is reported. <xref ref-type="fig" rid="pone-0087555-g002">Figure 2</xref>
indicates that Red-LDA produces topics that are more coherent than the Vanilla LDA ones.</p>
<fig id="pone-0087555-g002" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0087555.g002</object-id>
<label>Figure 2</label>
<caption><title>Comparison of Vanilla LDA and Red-LDA according to topic coherence.</title>
<p>Red-LDA produces more coherent topics than the baseline.</p>
</caption>
<graphic xlink:href="pone.0087555.g002"></graphic>
</fig>
</sec>
</sec>
<sec id="s2b"><title>Qualitative Analysis</title>
<p>To compare the quality of topics across models, we experimented with models learned with 50 topics, both for vanilla LDA and Red-LDA. Out of the 50 topics for each model, we fist aligned the topics learned using Jensen-Shannon divergence. The JS divergence identified 25 topic pairs with high enough similarity. First, the 25 topic pairs were presented to domain experts for review. For instance, the two clouds in <xref ref-type="fig" rid="pone-0087555-g003">Figure 3</xref>
, representative of a topic produced by Red-LDA and one produced by vanilla LDA, were shown side by side to the domain experts (along with the ranked list of words for each word cloud/topic). Other learned topics and their comparison to vanilla LDA are available at <ext-link ext-link-type="uri" xlink:href="http://www.cs.bgu.ac.il/~cohenrap/RedLDA">http://www.cs.bgu.ac.il/~cohenrap/RedLDA</ext-link>
. The experts were asked to chose which topic they felt was better quality, where a quality topic is defined both as representative of a clinical concept and as pure (<italic>i.e.</italic>
, with little words that did not belong to the concept represented by the topic). Second, the domain experts were presented with the remaining 50 individual topics (25 from vanilla LDA and 25 from Red-LDA), which were not aligned to any other topic according to their JS divergence. The experts were asked to rate on a Likert scale each topic’s quality according to the same quality criteria as for the pair comparison. The goal of the side-by-side comparison was to assess for the same topics, which model produced higher quality. The goal of the individual topic rating was to determine whether any model was able to identify high-quality topics, which the other model had missed (since they were not aligned to any other topic).</p>
<fig id="pone-0087555-g003" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0087555.g003</object-id>
<label>Figure 3</label>
<caption><title>Topics learnxled by Red-LDA (top) and Vanilla LDA (bottom) on the EHR corpus.</title>
<p>Both topics are about breast cancer (ca is an abbreviation for cancer). The Vanilla LDA topic, however, contains unrelated yet highly ranked words (<italic>e.g.</italic>
, eye, glaucoma, colonoscopy, albuterol). The Red-LDA topic was preferred by the domain experts.</p>
</caption>
<graphic xlink:href="pone.0087555.g003"></graphic>
</fig>
<p>When two domain experts compared the 25 aligned topic pairs, the Red-LDA topic was preferred in 13 cases, while the vanilla LDA topic was preferred in 4 cases. Furthermore, in the case of these four cases, the two experts disagreed in their assessment and expressed low confidence with their choice. When comparing the 25 non-aligned topics based on their likert score, on average Red-LDA topics were rated as higher quality than vanilla LDA topics 3.1 <italic>v.s.</italic>
2.7 (p = 0.1 on this relatively small sample). Overall, Red-LDA modeling tends to produce higher quality topics when interpreted in a clinical context.</p>
</sec>
</sec>
<sec id="s3"><title>Discussion</title>
<p>The experiments on different topic modeling techniques and their handling of the inherent redundancy of clinical notes indicate that while redundancy is harmful to traditional topic modeling technique, it is possible to mitigate its effect. Of all methods, Red-LDA yields the best topic models, both from a quantitative and a qualitative standpoint. Interestingly, the best-performing method does not discard any information, like the two baselines which outperform the vanilla LDA.</p>
<p>When examining the underlying data, we observe, for instance, that a very small number of patients in our data set had a particularly high number of notes (and thus high level of redundancy) and had both breast cancer and glaucoma (two unrelated clinical conditions). The vanilla LDA modeling was overwhelmed with the number of co-occurrences of the two conditions in the corpus overall because each note was considered equally, independently of whether they belonged to the same cluster of patient notes.</p>
<p>The three methods, DeleteDoc-LDA, DeleteWord-LDA, and Red-LDA, all rely on a preprocessing step to identify the “source” document in each record. There are many possible methods for choosing such a document (document size, metadata such as time stamps or average distance to the other documents within the cluster) or even to generate a projected source document to best represent the shared information between documents in the cluster. These methods and effects on different corpora may be studied in depth in the future.</p>
<p>As more and more corpora are made available to researchers, one can characterize them through different statistical properties (<italic>e.g.</italic>
, size of vocabulary, token-type ratio). One property which has received little explicit attention is the amount of redundancy inherent to a corpus. This research contributes primarily to the emerging field of understanding the characteristics of patient records, their impact on mining techniques, and how to correct for them <xref rid="pone.0087555-Hripcsak1" ref-type="bibr">[27]</xref>
.</p>
<p>Our findings on collection-level redundancy and text mining, are also pertinent to other domains. Redundancy, similar to the copy-and-paste redundancy of patient record, has been observed in Twitter <xref rid="pone.0087555-Zanzotto1" ref-type="bibr">[24]</xref>
(re-tweeting messages with only small changes), repeated spam messages which vary only slightly, and template texts, such as movie and theater listings <xref rid="pone.0087555-Mimno1" ref-type="bibr">[12]</xref>
. Redundancy occurs also in collections with longer documents. In the news domain, for instance, the same story authored by a news agency is reported by a number of sources, resulting in clusters of documents likely to be rich in redundancy <xref rid="pone.0087555-Barzilay1" ref-type="bibr">[28]</xref>
. Redundancy increases as events progress through time and stories get incrementally updated, very much like a patient record gets incrementally updated through time. As more and more social media emerges online, text mining methods must pay specific attention to the statistical parameters of text collection, and empirically analyze the implications of skewed distributions on data exploration methods. Our experiments indicate that (i) when string-level redundancy is prevalent, it has critical impact on the quality of topic modeling; and (ii) when handled within the topic learning, better quality topics can be produced than by ignoring the redundancy or discarding it harshly through pre-processing. This suggests, that not all redundancy is bad in text mining, and confirms that some redundancy indicate topic centrality and is thus important to keep during text mining activities, while some redundancy is just an artifact of documentation. We are interested in further understanding the relation between repetition and topical centrality across different domains in further experiments.</p>
</sec>
<sec sec-type="materials|methods" id="s4"><title>Materials and Methods</title>
<p>This section describes the data set on which we conducted our experiments, along with two baseline methods, that handle redundancy naively (Redundancy Elimination and DeleteLDA), and the algorithm for Red-LDA.</p>
<sec id="s4a"><title>Corpus of Patient Records</title>
<p>The dataset for our experiments is a large collection of patient records from the Columbia University Medical Center. IRB approval was obtained from the Columbia University Medical Center Institutional Review Board. As part of the approved IRB protocol, an application for waiver of authorization was filled and approved, as we investigate algorithms for learning statistical models across a very large population and historical data across many years, and thus obtaining individual consent would be impracticable. We collected the notes for 1,247 distinct patients. These 1,247 are all the patients who visit the outpatient clinic at the medical center and who also visit a nephrologist for kidney problems, and have overall at least three visits in their longitudinal record. As such, the data set represents a somewhat homogeneous set of patients from a clinical standpoint. In our experiments, we divided the corpus into a training set and held-out test set of 250 non-redundant documents from 250 patients not included in the training set. The assignment of the patients into the training or testing set was carried out randomly.</p>
<p>In the first visit to a physician, a patient note is authored in a “natural” generative process. Later notes, however, contain large sections of copy-paste material. The corpus we analyzed contains 8,557 notes, corresponding to 6M tokens. It contains notes from 1,247 distinct patients; therefore, most notes for the same patients are likely to share some redundancy with another note in the corpus. It is important to note, however, that there is a large amount of variation in the number of notes per patients (in other words, some patients are over-represented in the corpus compared to others).</p>
<p>To characterize the amount of redundancy present in the EHR corpus, we sampled same-patient notes and computed the Smith-Waterman alignment of the entire document. The average similarity between same-patient notes was 29%. 25% of the notes shared similarity of over 50% to another note in the corpus, confirming the notion that the EHR corpus is a naturally occuring, highly redundant corpus.</p>
</sec>
<sec id="s4b"><title>Input to Topic Modeling and Preprocessing</title>
<p>For all methods to learn a topic model, the input is a collection of patient records. A patient record is defined as a set of documents, or notes, written by the clinicians.</p>
<p>Given an input patient record in the collection of records, we use the online fingerprinting method described in <xref rid="pone.0087555-Cohen1" ref-type="bibr">[3]</xref>
to quantify the amount of redundancy in a record through aligning all strings in all documents in the record. The fingerprinting can thus identify in a record which strings are unique within the record and which are redundant throughout the record.</p>
</sec>
<sec id="s4c"><title>Two Baselines Through Redundancy Elimination</title>
<p>We present first two naïve approaches for solving the problem of redundancy in a given collection of records. Both methods rely on a fingerprint preprocessing. The first method, which we call DeleteDoc-LDA, simply removes the redundant documents from the corpus: for each patient record, a source document is chosen to represent the record as a whole, and the rest of the record is discarded. The second method, which we call DeleteWord-LDA, chooses a source to represent the cluster and deletes the redundant word from the other documents and adds these redundancy free documents to the corpus (<italic>i.e.</italic>
, only redundant portions of documents are discarded instead of the entire document like in the first method). In both methods, vanilla LDA is then applied to the resulting corpora to learn topics.</p>
<sec id="s4c1"><title>DeleteDoc-LDA</title>
<p>Given a training set of patient records, DeleteDoc-LDA consists of the following steps: (i) for each patient record in the training set, identify a source document that is representative of the record according to the fingerprinting preprocessing; (ii) discard all other documents in the record; and (iii) apply vanilla LDA to the source documents only.</p>
<p>The method of discarding all redundant documents is a drastic way to ensure that there is no redundancy in the input corpus to LDA. Since most of the redundant words in the corpus are removed, this solves the two problems we identified in the previous section of topic mismatch between identical tokens (produced from the same topic and copied) and skewed estimation of the weights of replicated words in different topics.</p>
<p>However, this approach has serious drawbacks if the resulting corpus is not large enough. For instance, if documents <inline-formula><inline-graphic xlink:href="pone.0087555.e015.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0087555.e016.jpg"></inline-graphic>
</inline-formula>
share a copy-pasted section of 30% of the two documents, removing one of the documents means losing the 70% non-redundant data in it. In addition, this method simply ignores the fact that segment repetition may be an indication of its topical relevance. The relation between repetition and importance has two aspects: when a segment is repeated, some of this repetition may be warranted by key terms, and reflect their importance. Other terms, however, within the repeated segment may simply be attached syntactically to the key terms, and their repetition is not indicative of their relevance. The second aspect is that, in certain domains like the clinical domain, repetition is not always an indication of topical centrality: the presence of copy-paste redundancy is an artifact of documentation practices and the software that clinicians use to author notes.</p>
</sec>
<sec id="s4c2"><title>DeleteWord-LDA</title>
<p>This method mitigates the redundancy by aligning the documents in the record containing the redundancy to the representative source document in the record, and then removing the copied words from the other documents (instead of whole documents, like in DeleteDoc-LDA). This solves the topic mismatch problem and the skewed estimation of weights since the redundant tokens are removed. However, it leaves the documents other than the resprentative missing some of the context of the document. If documents <inline-formula><inline-graphic xlink:href="pone.0087555.e017.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0087555.e018.jpg"></inline-graphic>
</inline-formula>
share a copy-pasted section of 30% of the documents, and <inline-formula><inline-graphic xlink:href="pone.0087555.e019.jpg"></inline-graphic>
</inline-formula>
is the represantative, we will delete 30% of <inline-formula><inline-graphic xlink:href="pone.0087555.e020.jpg"></inline-graphic>
</inline-formula>
which were copy-pasted. This may harm the topic estimation in <inline-formula><inline-graphic xlink:href="pone.0087555.e021.jpg"></inline-graphic>
</inline-formula>
, as the topic weigths in the new document do not include the removed sections.</p>
</sec>
</sec>
<sec id="s4d"><title>Redundancy-Aware LDA: Red-LDA</title>
<p>The approach for Red-LDA differs from the two previously described methods, as there is no data in the input records is discarded prior to learning topics. Instead, the redundancy is handled as part of the model.</p>
<p>Red-LDA is a probabilistic graphical model for generating a collection of records, which share different levels of redundancy. Like in LDA, each document is produced by a mixture of topics and each token is produced by one topic. Unlike LDA, Red-LDA assumes that each document belongs to a predefined cluster (<italic>i.e.</italic>
, a patient record) and some of its tokens are sampled from another document, the source document for the record, as identified by fingerprinting preprocessing step. The graphical model for Red-LDA is shown in <xref ref-type="fig" rid="pone-0087555-g004">Figure 4</xref>
.</p>
<fig id="pone-0087555-g004" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0087555.g004</object-id>
<label>Figure 4</label>
<caption><title>Red-LDA graphical model.</title>
<p>Patient records are collections of documents (plate C): of a source document and the rest of the record (M documents).</p>
</caption>
<graphic xlink:href="pone.0087555.g004"></graphic>
</fig>
<p>In each cluster of documents (or record) <inline-formula><inline-graphic xlink:href="pone.0087555.e022.jpg"></inline-graphic>
</inline-formula>
, one document <inline-formula><inline-graphic xlink:href="pone.0087555.e023.jpg"></inline-graphic>
</inline-formula>
is the original “source” document and the others <inline-formula><inline-graphic xlink:href="pone.0087555.e024.jpg"></inline-graphic>
</inline-formula>
are documents containing redundant parts copied from the source <inline-formula><inline-graphic xlink:href="pone.0087555.e025.jpg"></inline-graphic>
</inline-formula>
. Document <inline-formula><inline-graphic xlink:href="pone.0087555.e026.jpg"></inline-graphic>
</inline-formula>
is defined as a collection of tokens <inline-formula><inline-graphic xlink:href="pone.0087555.e027.jpg"></inline-graphic>
</inline-formula>
, where <inline-formula><inline-graphic xlink:href="pone.0087555.e028.jpg"></inline-graphic>
</inline-formula>
is the document length and each <inline-formula><inline-graphic xlink:href="pone.0087555.e029.jpg"></inline-graphic>
</inline-formula>
where <inline-formula><inline-graphic xlink:href="pone.0087555.e030.jpg"></inline-graphic>
</inline-formula>
is the vocabulary size. Topic <inline-formula><inline-graphic xlink:href="pone.0087555.e031.jpg"></inline-graphic>
</inline-formula>
is defined as <inline-formula><inline-graphic xlink:href="pone.0087555.e032.jpg"></inline-graphic>
</inline-formula>
where <inline-formula><inline-graphic xlink:href="pone.0087555.e033.jpg"></inline-graphic>
</inline-formula>
is the number of topics.</p>
<sec id="s4d1"><title>Generative process</title>
<p>The schematic description of the generative story is given in <xref ref-type="fig" rid="pone-0087555-g005">Figure 5</xref>
. To generate the corpus, we iterate over the patient records in the training set, each with a set of input documents. As such, the number of clusters is predefined, as is the number of documents in each cluster. For each cluster, we select one document as a source document. In our current implementation, the source document is selected as the longest note in the patient record. For the source document, its words are generated as per the standard LDA process – draw a multinomial distribution of topics for the document from a Dirichlet distribution; then for each word in the document, sample a topic from the document topic distribution, then sample a word from the topic. The change from LDA lies in the other documents in the cluster: within these documents, we first observe whether a word is copied from the source document in the cluster (this information comes from the fingerprinting pre-processing step). If the word is not copied, we generate it as usual by sampling a topic then the word from the topic. Else, we sample a topic from the document topic distribution, and then sample the source document for a word that has been assigned the same topic in the source document of the cluster.</p>
<fig id="pone-0087555-g005" orientation="portrait" position="float"><object-id pub-id-type="doi">10.1371/journal.pone.0087555.g005</object-id>
<label>Figure 5</label>
<caption><title>Red-LDA generative story.</title>
</caption>
<graphic xlink:href="pone.0087555.g005"></graphic>
</fig>
<p>By construction, the words that are copied from a source document (as determined by the fingerprinting step) are assigned the same topic in their copied occurrences as in the source document. Note also that because copied words must be assigned a topic that occurs in the source document, the distribution of topics in the copied documents is influenced by the distribution of topics in the source document (lines (14)-(17) in the generative story).</p>
</sec>
<sec id="s4d2"><title>The algorithm</title>
<p>Given a collection of patient records, each record consisting of a group of documents where one can be considered the source for each patient and the other the rest of the record. Given such a collection, the algorithm for Red-LDA is as follows. The graphical model is shown in <xref ref-type="fig" rid="pone-0087555-g004">Figure 4</xref>
. Sampling of a non-redundant word is the same as in the baseline Gibbs-Sampling for LDA:<disp-formula id="pone.0087555.e034"><graphic xlink:href="pone.0087555.e034.jpg" position="anchor" orientation="portrait"></graphic>
<label>(1)</label>
</disp-formula>
</p>
<p>When a redundant word is encountered, the topic is sampled from the topic distribution of identical tokens in the source document:</p>
<p><bold>Case</bold>
<inline-formula><inline-graphic xlink:href="pone.0087555.e035.jpg"></inline-graphic>
</inline-formula>
or <inline-formula><inline-graphic xlink:href="pone.0087555.e036.jpg"></inline-graphic>
</inline-formula>
:<disp-formula id="pone.0087555.e037"><graphic xlink:href="pone.0087555.e037.jpg" position="anchor" orientation="portrait"></graphic>
<label>(2)</label>
</disp-formula>
Where <inline-formula><inline-graphic xlink:href="pone.0087555.e038.jpg"></inline-graphic>
</inline-formula>
is like <inline-formula><inline-graphic xlink:href="pone.0087555.e039.jpg"></inline-graphic>
</inline-formula>
with the difference that words whose <inline-formula><inline-graphic xlink:href="pone.0087555.e040.jpg"></inline-graphic>
</inline-formula>
has the value <inline-formula><inline-graphic xlink:href="pone.0087555.e041.jpg"></inline-graphic>
</inline-formula>
(<italic>i.e.</italic>
, are copied) are not counted.</p>
<p><bold>Case</bold>
<inline-formula><inline-graphic xlink:href="pone.0087555.e042.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0087555.e043.jpg"></inline-graphic>
</inline-formula>
:<disp-formula id="pone.0087555.e044"><graphic xlink:href="pone.0087555.e044.jpg" position="anchor" orientation="portrait"></graphic>
<label>(3)</label>
</disp-formula>
Where <inline-formula><inline-graphic xlink:href="pone.0087555.e045.jpg"></inline-graphic>
</inline-formula>
is the probability of the word <inline-formula><inline-graphic xlink:href="pone.0087555.e046.jpg"></inline-graphic>
</inline-formula>
, which is known as having the topic <inline-formula><inline-graphic xlink:href="pone.0087555.e047.jpg"></inline-graphic>
</inline-formula>
, and being copied from the source document <inline-formula><inline-graphic xlink:href="pone.0087555.e048.jpg"></inline-graphic>
</inline-formula>
of the cluster <inline-formula><inline-graphic xlink:href="pone.0087555.e049.jpg"></inline-graphic>
</inline-formula>
where the word-topic pairs <inline-formula><inline-graphic xlink:href="pone.0087555.e050.jpg"></inline-graphic>
</inline-formula>
, <inline-formula><inline-graphic xlink:href="pone.0087555.e051.jpg"></inline-graphic>
</inline-formula>
have already been sampled. In our model we have:<disp-formula id="pone.0087555.e052"><graphic xlink:href="pone.0087555.e052.jpg" position="anchor" orientation="portrait"></graphic>
<label>(4)</label>
</disp-formula>
Where <inline-formula><inline-graphic xlink:href="pone.0087555.e053.jpg"></inline-graphic>
</inline-formula>
is the number of occurrences of the word <inline-formula><inline-graphic xlink:href="pone.0087555.e054.jpg"></inline-graphic>
</inline-formula>
assigned to topic <inline-formula><inline-graphic xlink:href="pone.0087555.e055.jpg"></inline-graphic>
</inline-formula>
in the source document <inline-formula><inline-graphic xlink:href="pone.0087555.e056.jpg"></inline-graphic>
</inline-formula>
of the cluster. The variable <inline-formula><inline-graphic xlink:href="pone.0087555.e057.jpg"></inline-graphic>
</inline-formula>
is a smoothing parameter with value between <inline-formula><inline-graphic xlink:href="pone.0087555.e058.jpg"></inline-graphic>
</inline-formula>
and <inline-formula><inline-graphic xlink:href="pone.0087555.e059.jpg"></inline-graphic>
</inline-formula>
.</p>
<p>The topic assignment for redundant words is not counted for the approximation of the topic-word multinomial distribution (in order to avoid giving a redundant word excess weight in the topic distribution, as described in the Problem Analysis Section).</p>
</sec>
<sec id="s4d3"><title>Running time</title>
<p>Red-LDA differs from vanilla LDA in two ways: (i) it requires pre-processing to identify source documents; and (ii) copied tokens are sampled from the source. The running time of the pre-processing step is very low <xref rid="pone.0087555-Cohen1" ref-type="bibr">[3]</xref>
(less than a minute on a standard PC for our dataset). The change during sampling ends up speeding up the LDA algorithm, as for redundant tokens we only sample their topic proportion in the source instead of using the topic-word and topic-document counts for all topics. Using 2,000 iterations for both vanilla LDA and Red-LDA on a standard PC for our dataset, vanilla LDA took 8.3 minutes, while Red-LDA took 7.35 minutes.</p>
</sec>
</sec>
<sec id="s4e"><title>Experimental Setup</title>
<p>The code for the fingerprinting preprocessing step is made publicly available at <ext-link ext-link-type="uri" xlink:href="https://sourceforge.net/projects/corpusredundanc">https://sourceforge.net/projects/corpusredundanc</ext-link>
. We used the Gibbs Sampling code by Heinrich <xref rid="pone.0087555-Heinrich1" ref-type="bibr">[29]</xref>
for the vanilla LDA. Red-LDA was implemented over the Heinrich code and is available at <ext-link ext-link-type="uri" xlink:href="https://sourceforge.net/projects/redlda">https://sourceforge.net/projects/redlda</ext-link>
. The fingerprinting preprocessing was applied to the training set of records, with a maximum redundancy threshold set to 15% for all methods, DeleteDoc-LDA, DeleteWord-LDA, and Red-LDA.</p>
</sec>
</sec>
</body>
<back><ref-list><title>References</title>
<ref id="pone.0087555-Siegler1"><label>1</label>
<mixed-citation publication-type="journal"><name><surname>Siegler</surname>
<given-names>E</given-names>
</name>
, <name><surname>Adelman</surname>
<given-names>R</given-names>
</name>
(<year>2009</year>
) <article-title>Copy and paste: a remediable hazard of electronic health records</article-title>
. <source>American Journal of Medicine</source>
<volume>122</volume>
: <fpage>495</fpage>
<pub-id pub-id-type="pmid">19486708</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Markel1"><label>2</label>
<mixed-citation publication-type="journal"><name><surname>Markel</surname>
<given-names>A</given-names>
</name>
(<year>2010</year>
) <article-title>Copy and paste of electronic health records: A modern medical illness</article-title>
. <source>American Journal of Medicine</source>
<volume>123</volume>
: <fpage>e9</fpage>
–<lpage>e9</lpage>
<pub-id pub-id-type="pmid">20399309</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Cohen1"><label>3</label>
<mixed-citation publication-type="journal"><name><surname>Cohen</surname>
<given-names>R</given-names>
</name>
, <name><surname>Elhadad</surname>
<given-names>M</given-names>
</name>
, <name><surname>Elhadad</surname>
<given-names>N</given-names>
</name>
(<year>2013</year>
) <article-title>Redundancy in electronic health record corpora: Analysis, impact on text mining performance and mitigation strategies</article-title>
. <source>BMC Bioinformatics</source>
<volume>14</volume>
: <fpage>10</fpage>
<pub-id pub-id-type="pmid">23323800</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Blei1"><label>4</label>
<mixed-citation publication-type="journal"><name><surname>Blei</surname>
<given-names>D</given-names>
</name>
, <name><surname>Ng</surname>
<given-names>A</given-names>
</name>
, <name><surname>Jordan</surname>
<given-names>M</given-names>
</name>
(<year>2003</year>
) <article-title>Latent Dirichlet allocation</article-title>
. <source>Journal of Machine Learning Research</source>
<volume>3</volume>
: <fpage>993</fpage>
–<lpage>1022</lpage>
</mixed-citation>
</ref>
<ref id="pone.0087555-Haghighi1"><label>5</label>
<mixed-citation publication-type="other">Haghighi A, Vanderwende L (2009) Exploring content models for multi-document summarization. In: Proc. HLT ACL. 326–379.</mixed-citation>
</ref>
<ref id="pone.0087555-Carterette1"><label>6</label>
<mixed-citation publication-type="other">Carterette B, Chandar P (2009) Probabilistic models of ranking novel documents for faceted topic retrieval. In: Proc. CIKM. 1287–1296.</mixed-citation>
</ref>
<ref id="pone.0087555-Blei2"><label>7</label>
<mixed-citation publication-type="other">Blei D, McAuliffe J (2007) Supervised topic models. arXiv preprint arXiv: 10030783.</mixed-citation>
</ref>
<ref id="pone.0087555-Ramage1"><label>8</label>
<mixed-citation publication-type="other">Ramage D, Hall D, Nallapati R, Manning C (2009) Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. In: Proc. EMNLP. 248–256.</mixed-citation>
</ref>
<ref id="pone.0087555-Wang1"><label>9</label>
<mixed-citation publication-type="journal"><name><surname>Wang</surname>
<given-names>H</given-names>
</name>
, <name><surname>Ding</surname>
<given-names>Y</given-names>
</name>
, <name><surname>Tang</surname>
<given-names>J</given-names>
</name>
, <name><surname>Dong</surname>
<given-names>X</given-names>
</name>
, <name><surname>He</surname>
<given-names>B</given-names>
</name>
, <etal>et al</etal>
(<year>2011</year>
) <article-title>Finding complex biological relationships in recent PubMed articles using Bio-LDA</article-title>
. <source>PloS ONE</source>
<volume>6</volume>
: <fpage>e17243</fpage>
<pub-id pub-id-type="pmid">21448266</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Walker1"><label>10</label>
<mixed-citation publication-type="other">Walker D, Lund W, Ringger E (2010) Evaluating models of latent document semantics in the presence of OCR errors. In: Proc. EMNLP. 240–250.</mixed-citation>
</ref>
<ref id="pone.0087555-Wallach1"><label>11</label>
<mixed-citation publication-type="journal"><name><surname>Wallach</surname>
<given-names>H</given-names>
</name>
, <name><surname>Murray</surname>
<given-names>I</given-names>
</name>
, <name><surname>Salakhutdinov</surname>
<given-names>R</given-names>
</name>
, <name><surname>Mimno</surname>
<given-names>D</given-names>
</name>
(<year>2009</year>
) <article-title>Evaluation methods for topic models</article-title>
. <source>In: Proc</source>
. <volume>ICML</volume>
: <fpage>1105</fpage>
–<lpage>1112</lpage>
</mixed-citation>
</ref>
<ref id="pone.0087555-Mimno1"><label>12</label>
<mixed-citation publication-type="other">Mimno D, Wallach HM, Talley E, Leenders M, McCallum A (2011) Optimizing semantic coherence in topic models. In: Proc. EMNLP. 262–272.</mixed-citation>
</ref>
<ref id="pone.0087555-Arnold1"><label>13</label>
<mixed-citation publication-type="other">Arnold C, El-Saden S, Bui A, Taira R (2010) Clinical case-based retrieval using latent topic analysis. In: Proc. AMIA. volume 2010, p. 26.</mixed-citation>
</ref>
<ref id="pone.0087555-SallebAouissi1"><label>14</label>
<mixed-citation publication-type="other">Salleb-Aouissi A, Radeva A, Passonneau R, Tomar A, Waltz D, <etal>et al</etal>
. (2011) Diving into a large corpus of pediatric notes. In: Proc. ICMLWorkshop on Learning from Unstructured Clinical Text.</mixed-citation>
</ref>
<ref id="pone.0087555-Saria1"><label>15</label>
<mixed-citation publication-type="other">Saria S, Koller D, Penn A (2010) Learning individual and population level traits from clinical temporal data. In: Proc. NIPS Predictive Models in Personalized Medicine Workshop.</mixed-citation>
</ref>
<ref id="pone.0087555-Perotte1"><label>16</label>
<mixed-citation publication-type="other">Perotte A, Bartlett N, Elhadad N, Wood F (2011) Hierarchically supervised latent Dirichlet allocation. In: Proc. NIPS. 2609–2617.</mixed-citation>
</ref>
<ref id="pone.0087555-Lehman1"><label>17</label>
<mixed-citation publication-type="other">Lehman LW, Saeed M, Long W, Lee J, Mark R (2012) Risk stratification of ICU patients using topic models inferred from unstructured progress notes. In: Proc. AMIA. 505–511.</mixed-citation>
</ref>
<ref id="pone.0087555-Bisgin1"><label>18</label>
<mixed-citation publication-type="journal"><name><surname>Bisgin</surname>
<given-names>H</given-names>
</name>
, <name><surname>Liu</surname>
<given-names>Z</given-names>
</name>
, <name><surname>Fang</surname>
<given-names>H</given-names>
</name>
, <name><surname>Xu</surname>
<given-names>X</given-names>
</name>
, <name><surname>Tong</surname>
<given-names>W</given-names>
</name>
(<year>2011</year>
) <article-title>Mining FDA drug labels using an unsupervised learning technique-topic modeling</article-title>
. <source>BMC Bioinformatics</source>
<volume>12</volume>
: <fpage>S11</fpage>
<pub-id pub-id-type="pmid">22166012</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Lippincott1"><label>19</label>
<mixed-citation publication-type="journal"><name><surname>Lippincott</surname>
<given-names>T</given-names>
</name>
, <name><surname>Séaghdha</surname>
<given-names>DÓ</given-names>
</name>
, <name><surname>Korhonen</surname>
<given-names>A</given-names>
</name>
(<year>2011</year>
) <article-title>Exploring subdomain variation in biomedical language</article-title>
. <source>BMC Bioinformatics</source>
<volume>12</volume>
: <fpage>212</fpage>
<pub-id pub-id-type="pmid">21619603</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Bisgin2"><label>20</label>
<mixed-citation publication-type="journal"><name><surname>Bisgin</surname>
<given-names>H</given-names>
</name>
, <name><surname>Liu</surname>
<given-names>Z</given-names>
</name>
, <name><surname>Kelly</surname>
<given-names>R</given-names>
</name>
, <name><surname>Fang</surname>
<given-names>H</given-names>
</name>
, <name><surname>Xu</surname>
<given-names>X</given-names>
</name>
, <etal>et al</etal>
(<year>2012</year>
) <article-title>Investigating drug repositioning opportunities in FDA drug labels through topic modeling</article-title>
. <source>BMC Bioinformatics</source>
<volume>13</volume>
: <fpage>S6</fpage>
<pub-id pub-id-type="pmid">23046522</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Crain1"><label>21</label>
<mixed-citation publication-type="other">Crain S, Yang S, Zha H, Jiao Y (2010) Dialect topic modeling for improved consumer medical search. In: Proc. AMIA. 132–136.</mixed-citation>
</ref>
<ref id="pone.0087555-Arnold2"><label>22</label>
<mixed-citation publication-type="other">Arnold C, Speier W (2012) A topic model of clinical reports. In: Proc. SIGIR. 1031–1032.</mixed-citation>
</ref>
<ref id="pone.0087555-Halpern1"><label>23</label>
<mixed-citation publication-type="other">Halpern Y, Horng S, Nathanson L, Shapiro N, Sontag D (2012) A comparison of dimensionality reduction techniques for unstructured clinical text. In: Proc. ICML Workshop on Clinical Data Analysis.</mixed-citation>
</ref>
<ref id="pone.0087555-Zanzotto1"><label>24</label>
<mixed-citation publication-type="other">Zanzotto F, Pennacchiotti M, Tsioutsiouliklis K (2011) Linguistic redundancy in Twitter. In: Proc. EMNLP. 659–669.</mixed-citation>
</ref>
<ref id="pone.0087555-Holm1"><label>25</label>
<mixed-citation publication-type="journal"><name><surname>Holm</surname>
<given-names>L</given-names>
</name>
, <name><surname>Sander</surname>
<given-names>C</given-names>
</name>
(<year>1993</year>
) <article-title>Protein structure comparison by alignment of distance matrices</article-title>
. <source>Journal of molecular biology</source>
<volume>233</volume>
: <fpage>123</fpage>
–<lpage>138</lpage>
<pub-id pub-id-type="pmid">8377180</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Buslje1"><label>26</label>
<mixed-citation publication-type="journal"><name><surname>Buslje</surname>
<given-names>C</given-names>
</name>
, <name><surname>Santos</surname>
<given-names>J</given-names>
</name>
, <name><surname>Delfino</surname>
<given-names>J</given-names>
</name>
, <name><surname>Nielsen</surname>
<given-names>M</given-names>
</name>
(<year>2009</year>
) <article-title>Correction for phylogeny, small number of observations and data redundancy improves the identification of coevolving amino acid pairs using mutual information</article-title>
. <source>Bioinformatics</source>
<volume>25</volume>
: <fpage>1125</fpage>
–<lpage>1131</lpage>
<pub-id pub-id-type="pmid">19276150</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Hripcsak1"><label>27</label>
<mixed-citation publication-type="journal"><name><surname>Hripcsak</surname>
<given-names>G</given-names>
</name>
, <name><surname>Albers</surname>
<given-names>D</given-names>
</name>
(<year>2013</year>
) <article-title>Next-generation phenotyping of electronic health records</article-title>
. <source>Journal of the American Medical Informatics Association</source>
<volume>20</volume>
: <fpage>117</fpage>
–<lpage>121</lpage>
<pub-id pub-id-type="pmid">22955496</pub-id>
</mixed-citation>
</ref>
<ref id="pone.0087555-Barzilay1"><label>28</label>
<mixed-citation publication-type="other">Barzilay R, McKeown K, Elhadad M (1999) Information fusion in the context of multi-document summarization. In: Proc. ACL. 550–557.</mixed-citation>
</ref>
<ref id="pone.0087555-Heinrich1"><label>29</label>
<mixed-citation publication-type="other">Heinrich G (2008) Parameter estimation for text analysis. Technical report, University of Leipzig.</mixed-citation>
</ref>
</ref-list>
</back>
</pmc>
<affiliations><list><country><li>Israël</li>
<li>États-Unis</li>
</country>
<region><li>État de New York</li>
</region>
<settlement><li>New York</li>
</settlement>
<orgName><li>Université Columbia</li>
</orgName>
</list>
<tree><country name="Israël"><noRegion><name sortKey="Cohen, Raphael" sort="Cohen, Raphael" uniqKey="Cohen R" first="Raphael" last="Cohen">Raphael Cohen</name>
</noRegion>
<name sortKey="Aviram, Iddo" sort="Aviram, Iddo" uniqKey="Aviram I" first="Iddo" last="Aviram">Iddo Aviram</name>
<name sortKey="Elhadad, Michael" sort="Elhadad, Michael" uniqKey="Elhadad M" first="Michael" last="Elhadad">Michael Elhadad</name>
</country>
<country name="États-Unis"><region name="État de New York"><name sortKey="Elhadad, Noemie" sort="Elhadad, Noemie" uniqKey="Elhadad N" first="Noémie" last="Elhadad">Noémie Elhadad</name>
</region>
</country>
</tree>
</affiliations>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/OcrV1/Data/Pmc/Checkpoint
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000042 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Checkpoint/biblio.hfd -nk 000042 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Ticri/CIDE |area= OcrV1 |flux= Pmc |étape= Checkpoint |type= RBID |clé= PMC:3923754 |texte= Redundancy-Aware Topic Modeling for Patient Record Notes }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Checkpoint/RBID.i -Sk "pubmed:24551060" \ | HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Checkpoint/biblio.hfd \ | NlmPubMed2Wicri -a OcrV1
![]() | This area was generated with Dilib version V0.6.32. | ![]() |