Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department
Identifieur interne : 000169 ( Pmc/Corpus ); précédent : 000168; suivant : 000170Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department
Auteurs : Samuel M. Brown ; Jason Jones ; Kathryn Gibb Kuttler ; Roger K. Keddington ; Todd L. Allen ; Peter HaugSource :
- BMC Emergency Medicine [ 1471-227X ] ; 2016.
Abstract
Sepsis is an often-fatal syndrome resulting from severe infection. Rapid identification and treatment are critical for septic patients. We therefore developed a probabilistic model to identify septic patients in the emergency department (ED). We aimed to produce a model that identifies 80 % of sepsis patients, with no more than 15 false positive alerts per day, within one hour of ED admission, using routine clinical data.
We developed the model using retrospective data for 132,748 ED encounters (549 septic), with manual chart review to confirm cases of severe sepsis or septic shock from January 2006 through December 2008. A naïve Bayes model was used to select model features, starting with clinician-proposed candidate variables, which were then used to calculate the probability of sepsis. We evaluated the accuracy of the resulting model in 93,733 ED encounters from April 2009 through June 2010.
The final model included mean blood pressure, temperature, age, heart rate, and white blood cell count. The area under the receiver operating characteristic curve (AUC) for the continuous predictor model was 0.953. The binary alert achieved 76.4 % sensitivity with a false positive rate of 4.7 %.
We developed and validated a probabilistic model to identify sepsis early in an ED encounter. Despite changes in process, organizational focus, and the H1N1 influenza pandemic, our model performed adequately in our validation cohort, suggesting that it will be generalizable.
Url:
DOI: 10.1186/s12873-016-0095-0
PubMed: 27549755
PubMed Central: 4994262
Links to Exploration step
PMC:4994262Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department</title>
<author><name sortKey="Brown, Samuel M" sort="Brown, Samuel M" uniqKey="Brown S" first="Samuel M." last="Brown">Samuel M. Brown</name>
<affiliation><nlm:aff id="Aff1">Pulmonary and Critical Care Medicine, Intermountain Medical Center, Murray, USA</nlm:aff>
</affiliation>
<affiliation><nlm:aff id="Aff2">Pulmonary and Critical Care Medicine, University of Utah School of Medicine, Salt Lake City, USA</nlm:aff>
</affiliation>
<affiliation><nlm:aff id="Aff7">Shock Trauma ICU, Intermountain Medical Center, 5121 S. Cottonwood Street, Murray, UT 84107 USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Jones, Jason" sort="Jones, Jason" uniqKey="Jones J" first="Jason" last="Jones">Jason Jones</name>
<affiliation><nlm:aff id="Aff3">Clinical Intelligence and Decision Support, Kaiser Permanente Southern California, Pasadena, USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Kuttler, Kathryn Gibb" sort="Kuttler, Kathryn Gibb" uniqKey="Kuttler K" first="Kathryn Gibb" last="Kuttler">Kathryn Gibb Kuttler</name>
<affiliation><nlm:aff id="Aff1">Pulmonary and Critical Care Medicine, Intermountain Medical Center, Murray, USA</nlm:aff>
</affiliation>
<affiliation><nlm:aff id="Aff4">Homer Warner Center for Informatics Research, Intermountain Healthcare, Murray, USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Keddington, Roger K" sort="Keddington, Roger K" uniqKey="Keddington R" first="Roger K." last="Keddington">Roger K. Keddington</name>
<affiliation><nlm:aff id="Aff5">Intensive Medicine Clinical Program, Intermountain Healthcare, Salt Lake City, USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Allen, Todd L" sort="Allen, Todd L" uniqKey="Allen T" first="Todd L." last="Allen">Todd L. Allen</name>
<affiliation><nlm:aff id="Aff6">Institute for Healthcare Delivery Research, Intermountain Healthcare, Salt Lake City, USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Haug, Peter" sort="Haug, Peter" uniqKey="Haug P" first="Peter" last="Haug">Peter Haug</name>
<affiliation><nlm:aff id="Aff4">Homer Warner Center for Informatics Research, Intermountain Healthcare, Murray, USA</nlm:aff>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PMC</idno>
<idno type="pmid">27549755</idno>
<idno type="pmc">4994262</idno>
<idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4994262</idno>
<idno type="RBID">PMC:4994262</idno>
<idno type="doi">10.1186/s12873-016-0095-0</idno>
<date when="2016">2016</date>
<idno type="wicri:Area/Pmc/Corpus">000169</idno>
<idno type="wicri:explorRef" wicri:stream="Pmc" wicri:step="Corpus" wicri:corpus="PMC">000169</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department</title>
<author><name sortKey="Brown, Samuel M" sort="Brown, Samuel M" uniqKey="Brown S" first="Samuel M." last="Brown">Samuel M. Brown</name>
<affiliation><nlm:aff id="Aff1">Pulmonary and Critical Care Medicine, Intermountain Medical Center, Murray, USA</nlm:aff>
</affiliation>
<affiliation><nlm:aff id="Aff2">Pulmonary and Critical Care Medicine, University of Utah School of Medicine, Salt Lake City, USA</nlm:aff>
</affiliation>
<affiliation><nlm:aff id="Aff7">Shock Trauma ICU, Intermountain Medical Center, 5121 S. Cottonwood Street, Murray, UT 84107 USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Jones, Jason" sort="Jones, Jason" uniqKey="Jones J" first="Jason" last="Jones">Jason Jones</name>
<affiliation><nlm:aff id="Aff3">Clinical Intelligence and Decision Support, Kaiser Permanente Southern California, Pasadena, USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Kuttler, Kathryn Gibb" sort="Kuttler, Kathryn Gibb" uniqKey="Kuttler K" first="Kathryn Gibb" last="Kuttler">Kathryn Gibb Kuttler</name>
<affiliation><nlm:aff id="Aff1">Pulmonary and Critical Care Medicine, Intermountain Medical Center, Murray, USA</nlm:aff>
</affiliation>
<affiliation><nlm:aff id="Aff4">Homer Warner Center for Informatics Research, Intermountain Healthcare, Murray, USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Keddington, Roger K" sort="Keddington, Roger K" uniqKey="Keddington R" first="Roger K." last="Keddington">Roger K. Keddington</name>
<affiliation><nlm:aff id="Aff5">Intensive Medicine Clinical Program, Intermountain Healthcare, Salt Lake City, USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Allen, Todd L" sort="Allen, Todd L" uniqKey="Allen T" first="Todd L." last="Allen">Todd L. Allen</name>
<affiliation><nlm:aff id="Aff6">Institute for Healthcare Delivery Research, Intermountain Healthcare, Salt Lake City, USA</nlm:aff>
</affiliation>
</author>
<author><name sortKey="Haug, Peter" sort="Haug, Peter" uniqKey="Haug P" first="Peter" last="Haug">Peter Haug</name>
<affiliation><nlm:aff id="Aff4">Homer Warner Center for Informatics Research, Intermountain Healthcare, Murray, USA</nlm:aff>
</affiliation>
</author>
</analytic>
<series><title level="j">BMC Emergency Medicine</title>
<idno type="eISSN">1471-227X</idno>
<imprint><date when="2016">2016</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en"><sec><title>Background</title>
<p>Sepsis is an often-fatal syndrome resulting from severe infection. Rapid identification and treatment are critical for septic patients. We therefore developed a probabilistic model to identify septic patients in the emergency department (ED). We aimed to produce a model that identifies 80 % of sepsis patients, with no more than 15 false positive alerts per day, within one hour of ED admission, using routine clinical data.</p>
</sec>
<sec><title>Methods</title>
<p>We developed the model using retrospective data for 132,748 ED encounters (549 septic), with manual chart review to confirm cases of severe sepsis or septic shock from January 2006 through December 2008. A naïve Bayes model was used to select model features, starting with clinician-proposed candidate variables, which were then used to calculate the probability of sepsis. We evaluated the accuracy of the resulting model in 93,733 ED encounters from April 2009 through June 2010.</p>
</sec>
<sec><title>Results</title>
<p>The final model included mean blood pressure, temperature, age, heart rate, and white blood cell count. The area under the receiver operating characteristic curve (AUC) for the continuous predictor model was 0.953. The binary alert achieved 76.4 % sensitivity with a false positive rate of 4.7 %.</p>
</sec>
<sec><title>Conclusions</title>
<p>We developed and validated a probabilistic model to identify sepsis early in an ED encounter. Despite changes in process, organizational focus, and the H1N1 influenza pandemic, our model performed adequately in our validation cohort, suggesting that it will be generalizable.</p>
</sec>
</div>
</front>
<back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Angus, Dc" uniqKey="Angus D">DC Angus</name>
</author>
<author><name sortKey="Linde Zwirble, Wt" uniqKey="Linde Zwirble W">WT Linde-Zwirble</name>
</author>
<author><name sortKey="Lidicker, J" uniqKey="Lidicker J">J Lidicker</name>
</author>
<author><name sortKey="Clermont, G" uniqKey="Clermont G">G Clermont</name>
</author>
<author><name sortKey="Carcillo, J" uniqKey="Carcillo J">J Carcillo</name>
</author>
<author><name sortKey="Pinsky, Mr" uniqKey="Pinsky M">MR Pinsky</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Martin, Gs" uniqKey="Martin G">GS Martin</name>
</author>
<author><name sortKey="Mannino, Dm" uniqKey="Mannino D">DM Mannino</name>
</author>
<author><name sortKey="Eaton, S" uniqKey="Eaton S">S Eaton</name>
</author>
<author><name sortKey="Moss, M" uniqKey="Moss M">M Moss</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Rivers, E" uniqKey="Rivers E">E Rivers</name>
</author>
<author><name sortKey="Nguyen, B" uniqKey="Nguyen B">B Nguyen</name>
</author>
<author><name sortKey="Havstad, S" uniqKey="Havstad S">S Havstad</name>
</author>
<author><name sortKey="Ressler, J" uniqKey="Ressler J">J Ressler</name>
</author>
<author><name sortKey="Muzzin, A" uniqKey="Muzzin A">A Muzzin</name>
</author>
<author><name sortKey="Knoblich, B" uniqKey="Knoblich B">B Knoblich</name>
</author>
<author><name sortKey="Peterson, E" uniqKey="Peterson E">E Peterson</name>
</author>
<author><name sortKey="Tomlanovich, M" uniqKey="Tomlanovich M">M Tomlanovich</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Yealy, Dm" uniqKey="Yealy D">DM Yealy</name>
</author>
<author><name sortKey="Kellum, Ja" uniqKey="Kellum J">JA Kellum</name>
</author>
<author><name sortKey="Huang, Dt" uniqKey="Huang D">DT Huang</name>
</author>
<author><name sortKey="Barnato, Ae" uniqKey="Barnato A">AE Barnato</name>
</author>
<author><name sortKey="Weissfeld, La" uniqKey="Weissfeld L">LA Weissfeld</name>
</author>
<author><name sortKey="Pike, F" uniqKey="Pike F">F Pike</name>
</author>
<author><name sortKey="Terndrup, T" uniqKey="Terndrup T">T Terndrup</name>
</author>
<author><name sortKey="Wang, He" uniqKey="Wang H">HE Wang</name>
</author>
<author><name sortKey="Hou, Pc" uniqKey="Hou P">PC Hou</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Lilly, Cm" uniqKey="Lilly C">CM Lilly</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Dellinger, Rp" uniqKey="Dellinger R">RP Dellinger</name>
</author>
<author><name sortKey="Levy, Mm" uniqKey="Levy M">MM Levy</name>
</author>
<author><name sortKey="Rhodes, A" uniqKey="Rhodes A">A Rhodes</name>
</author>
<author><name sortKey="Annane, D" uniqKey="Annane D">D Annane</name>
</author>
<author><name sortKey="Gerlach, H" uniqKey="Gerlach H">H Gerlach</name>
</author>
<author><name sortKey="Opal, Sm" uniqKey="Opal S">SM Opal</name>
</author>
<author><name sortKey="Sevransky, Je" uniqKey="Sevransky J">JE Sevransky</name>
</author>
<author><name sortKey="Sprung, Cl" uniqKey="Sprung C">CL Sprung</name>
</author>
<author><name sortKey="Douglas, Is" uniqKey="Douglas I">IS Douglas</name>
</author>
<author><name sortKey="Jaeschke, R" uniqKey="Jaeschke R">R Jaeschke</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Jones, Jp" uniqKey="Jones J">JP Jones</name>
</author>
<author><name sortKey="Kuttler, K" uniqKey="Kuttler K">K Kuttler</name>
</author>
<author><name sortKey="Keddington, Rk" uniqKey="Keddington R">RK Keddington</name>
</author>
<author><name sortKey="Allen, Tl" uniqKey="Allen T">TL Allen</name>
</author>
<author><name sortKey="Brown, Sm" uniqKey="Brown S">SM Brown</name>
</author>
<author><name sortKey="Haug, Pj" uniqKey="Haug P">PJ Haug</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Miller, Rr" uniqKey="Miller R">RR Miller</name>
</author>
<author><name sortKey="Dong, L" uniqKey="Dong L">L Dong</name>
</author>
<author><name sortKey="Nelson, Nc" uniqKey="Nelson N">NC Nelson</name>
</author>
<author><name sortKey="Brown, Sm" uniqKey="Brown S">SM Brown</name>
</author>
<author><name sortKey="Kuttler, Kg" uniqKey="Kuttler K">KG Kuttler</name>
</author>
<author><name sortKey="Probst, Dr" uniqKey="Probst D">DR Probst</name>
</author>
<author><name sortKey="Allen, Tl" uniqKey="Allen T">TL Allen</name>
</author>
<author><name sortKey="Clemmer, Tp" uniqKey="Clemmer T">TP Clemmer</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Bone, Rc" uniqKey="Bone R">RC Bone</name>
</author>
<author><name sortKey="Balk, Ra" uniqKey="Balk R">RA Balk</name>
</author>
<author><name sortKey="Cerra, Fb" uniqKey="Cerra F">FB Cerra</name>
</author>
<author><name sortKey="Dellinger, Rp" uniqKey="Dellinger R">RP Dellinger</name>
</author>
<author><name sortKey="Fein, Am" uniqKey="Fein A">AM Fein</name>
</author>
<author><name sortKey="Knaus, Wa" uniqKey="Knaus W">WA Knaus</name>
</author>
<author><name sortKey="Schein, Rm" uniqKey="Schein R">RM Schein</name>
</author>
<author><name sortKey="Sibbald, Wj" uniqKey="Sibbald W">WJ Sibbald</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Miller, Rr" uniqKey="Miller R">RR Miller</name>
</author>
<author><name sortKey="Markewitz, Ba" uniqKey="Markewitz B">BA Markewitz</name>
</author>
<author><name sortKey="Rolfs, Rt" uniqKey="Rolfs R">RT Rolfs</name>
</author>
<author><name sortKey="Brown, Sm" uniqKey="Brown S">SM Brown</name>
</author>
<author><name sortKey="Dascomb, Kk" uniqKey="Dascomb K">KK Dascomb</name>
</author>
<author><name sortKey="Grissom, Ck" uniqKey="Grissom C">CK Grissom</name>
</author>
<author><name sortKey="Friedrichs, Md" uniqKey="Friedrichs M">MD Friedrichs</name>
</author>
<author><name sortKey="Mayer, J" uniqKey="Mayer J">J Mayer</name>
</author>
<author><name sortKey="Hirshberg, El" uniqKey="Hirshberg E">EL Hirshberg</name>
</author>
<author><name sortKey="Conklin, J" uniqKey="Conklin J">J Conklin</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Harrell, Fe" uniqKey="Harrell F">FE Harrell</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Efron, B" uniqKey="Efron B">B Efron</name>
</author>
<author><name sortKey="Tibshirani, R" uniqKey="Tibshirani R">R Tibshirani</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Van Der Sijs, H" uniqKey="Van Der Sijs H">H van der Sijs</name>
</author>
<author><name sortKey="Aarts, J" uniqKey="Aarts J">J Aarts</name>
</author>
<author><name sortKey="Vulto, A" uniqKey="Vulto A">A Vulto</name>
</author>
<author><name sortKey="Berg, M" uniqKey="Berg M">M Berg</name>
</author>
</analytic>
</biblStruct>
</listBibl>
</div1>
</back>
</TEI>
<pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">BMC Emerg Med</journal-id>
<journal-id journal-id-type="iso-abbrev">BMC Emerg Med</journal-id>
<journal-title-group><journal-title>BMC Emergency Medicine</journal-title>
</journal-title-group>
<issn pub-type="epub">1471-227X</issn>
<publisher><publisher-name>BioMed Central</publisher-name>
<publisher-loc>London</publisher-loc>
</publisher>
</journal-meta>
<article-meta><article-id pub-id-type="pmid">27549755</article-id>
<article-id pub-id-type="pmc">4994262</article-id>
<article-id pub-id-type="publisher-id">95</article-id>
<article-id pub-id-type="doi">10.1186/s12873-016-0095-0</article-id>
<article-categories><subj-group subj-group-type="heading"><subject>Research Article</subject>
</subj-group>
</article-categories>
<title-group><article-title>Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department</article-title>
</title-group>
<contrib-group><contrib contrib-type="author" corresp="yes" equal-contrib="yes"><name><surname>Brown</surname>
<given-names>Samuel M.</given-names>
</name>
<address><phone>(801) 507-6529</phone>
<email>samuel.brown@imail.org</email>
</address>
<xref ref-type="aff" rid="Aff1">1</xref>
<xref ref-type="aff" rid="Aff2">2</xref>
<xref ref-type="aff" rid="Aff7">7</xref>
</contrib>
<contrib contrib-type="author" equal-contrib="yes"><name><surname>Jones</surname>
<given-names>Jason</given-names>
</name>
<xref ref-type="aff" rid="Aff3">3</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Kuttler</surname>
<given-names>Kathryn Gibb</given-names>
</name>
<xref ref-type="aff" rid="Aff1">1</xref>
<xref ref-type="aff" rid="Aff4">4</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Keddington</surname>
<given-names>Roger K.</given-names>
</name>
<xref ref-type="aff" rid="Aff5">5</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Allen</surname>
<given-names>Todd L.</given-names>
</name>
<xref ref-type="aff" rid="Aff6">6</xref>
</contrib>
<contrib contrib-type="author"><name><surname>Haug</surname>
<given-names>Peter</given-names>
</name>
<xref ref-type="aff" rid="Aff4">4</xref>
</contrib>
<aff id="Aff1"><label>1</label>
Pulmonary and Critical Care Medicine, Intermountain Medical Center, Murray, USA</aff>
<aff id="Aff2"><label>2</label>
Pulmonary and Critical Care Medicine, University of Utah School of Medicine, Salt Lake City, USA</aff>
<aff id="Aff3"><label>3</label>
Clinical Intelligence and Decision Support, Kaiser Permanente Southern California, Pasadena, USA</aff>
<aff id="Aff4"><label>4</label>
Homer Warner Center for Informatics Research, Intermountain Healthcare, Murray, USA</aff>
<aff id="Aff5"><label>5</label>
Intensive Medicine Clinical Program, Intermountain Healthcare, Salt Lake City, USA</aff>
<aff id="Aff6"><label>6</label>
Institute for Healthcare Delivery Research, Intermountain Healthcare, Salt Lake City, USA</aff>
<aff id="Aff7"><label>7</label>
Shock Trauma ICU, Intermountain Medical Center, 5121 S. Cottonwood Street, Murray, UT 84107 USA</aff>
</contrib-group>
<pub-date pub-type="epub"><day>22</day>
<month>8</month>
<year>2016</year>
</pub-date>
<pub-date pub-type="pmc-release"><day>22</day>
<month>8</month>
<year>2016</year>
</pub-date>
<pub-date pub-type="collection"><year>2016</year>
</pub-date>
<volume>16</volume>
<issue>1</issue>
<elocation-id>31</elocation-id>
<history><date date-type="received"><day>5</day>
<month>3</month>
<year>2016</year>
</date>
<date date-type="accepted"><day>12</day>
<month>8</month>
<year>2016</year>
</date>
</history>
<permissions><copyright-statement>© The Author(s). 2016</copyright-statement>
<license license-type="OpenAccess"><license-p><bold>Open Access</bold>
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0/">http://creativecommons.org/licenses/by/4.0/</ext-link>
), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/publicdomain/zero/1.0/">http://creativecommons.org/publicdomain/zero/1.0/</ext-link>
) applies to the data made available in this article, unless otherwise stated.</license-p>
</license>
</permissions>
<abstract id="Abs1"><sec><title>Background</title>
<p>Sepsis is an often-fatal syndrome resulting from severe infection. Rapid identification and treatment are critical for septic patients. We therefore developed a probabilistic model to identify septic patients in the emergency department (ED). We aimed to produce a model that identifies 80 % of sepsis patients, with no more than 15 false positive alerts per day, within one hour of ED admission, using routine clinical data.</p>
</sec>
<sec><title>Methods</title>
<p>We developed the model using retrospective data for 132,748 ED encounters (549 septic), with manual chart review to confirm cases of severe sepsis or septic shock from January 2006 through December 2008. A naïve Bayes model was used to select model features, starting with clinician-proposed candidate variables, which were then used to calculate the probability of sepsis. We evaluated the accuracy of the resulting model in 93,733 ED encounters from April 2009 through June 2010.</p>
</sec>
<sec><title>Results</title>
<p>The final model included mean blood pressure, temperature, age, heart rate, and white blood cell count. The area under the receiver operating characteristic curve (AUC) for the continuous predictor model was 0.953. The binary alert achieved 76.4 % sensitivity with a false positive rate of 4.7 %.</p>
</sec>
<sec><title>Conclusions</title>
<p>We developed and validated a probabilistic model to identify sepsis early in an ED encounter. Despite changes in process, organizational focus, and the H1N1 influenza pandemic, our model performed adequately in our validation cohort, suggesting that it will be generalizable.</p>
</sec>
</abstract>
<kwd-group xml:lang="en"><title>Keywords</title>
<kwd>Sepsis</kwd>
<kwd>Automated detection</kwd>
<kwd>Bayesian classifier</kwd>
</kwd-group>
<funding-group><award-group><funding-source><institution-wrap><institution-id institution-id-type="FundRef">http://dx.doi.org/10.13039/100000057</institution-id>
<institution>National Institute of General Medical Sciences</institution>
</institution-wrap>
</funding-source>
<award-id>k23gm094465</award-id>
<principal-award-recipient><name><surname>Brown</surname>
<given-names>Samuel M.</given-names>
</name>
</principal-award-recipient>
</award-group>
</funding-group>
<custom-meta-group><custom-meta><meta-name>issue-copyright-statement</meta-name>
<meta-value>© The Author(s) 2016</meta-value>
</custom-meta>
</custom-meta-group>
</article-meta>
</front>
<body><sec id="Sec1"><title>Background</title>
<p>Sepsis, especially in its more serious forms, severe sepsis and septic shock, is an important clinical and public health problem, accounting for approximately 750,000 hospital admissions in the USA annually and 20 % of intensive care unit (ICU) admissions. Furthermore, severe sepsis and septic shock are associated with high short-term mortality (18–40 %) [<xref ref-type="bibr" rid="CR1">1</xref>
, <xref ref-type="bibr" rid="CR2">2</xref>
]. Rapid identification and initiation of treatment are considered critical for patient survival with sepsis, even if some controversy exists about the precise details of treatment protocols [<xref ref-type="bibr" rid="CR3">3</xref>
–<xref ref-type="bibr" rid="CR5">5</xref>
]. Early, appropriate treatment of sepsis is widely recommended by the academic societies and has long been advocated by the inter-society Surviving Sepsis Campaign [<xref ref-type="bibr" rid="CR6">6</xref>
].</p>
<sec id="Sec2"><title>Importance</title>
<p>Given the limited time during which to affect the course of sepsis, early identification of severe sepsis or septic shock is crucial. Early, automated identification of severe sepsis or septic shock could potentially a) decrease the time to advanced diagnostics and risk assessment, b) decrease time to receipt of appropriate treatment, such as antibiotics, volume expansion, and vasopressors, c) enhance quality improvement efforts, and d) assist with timely study enrollment for septic patients.</p>
</sec>
<sec id="Sec3"><title>Goals of this investigation</title>
<p>In order to facilitate early identification, we sought to develop a probabilistic model to identify septic patients within one hour of entering the emergency department (ED) as part of a broader effort to improve compliance with Surviving Sepsis Campaign sepsis management bundles at Intermountain hospitals [<xref ref-type="bibr" rid="CR7">7</xref>
]. We aimed to develop a system that demonstrated good sensitivity with a clinically tolerable false positive rate.</p>
</sec>
</sec>
<sec id="Sec4"><title>Methods</title>
<sec id="Sec5"><title>Study design and setting</title>
<p>As we reported in abstract form, we developed the model using a data set that included all Emergency Department (ED) encounters (132,748 encounters, of which 549 were septic) from January 2006 through December 2008 from both LDS Hospital and Intermountain Medical Center [<xref ref-type="bibr" rid="CR8">8</xref>
]. In that retrospective study, data available within 1 h of ED arrival (temperature, mean blood pressure, heart rate, and white blood cell count) were used for a model with an area under the curve (AUC) of 0.836. The prospective validation we report here covered April 2009 through June 2010 and was conducted at Intermountain Medical Center. Intermountain Medical Center is a 456-bed teaching hospital affiliated with the University of Utah School of Medicine which replaced LDS Hospital in October 2007 as Intermountain Healthcare’s tertiary referral center and Level One trauma facility. (LDS Hospital remained in operation, transitioning into a secondary referral center status.)</p>
</sec>
<sec id="Sec6"><title>Selection of participants</title>
<p>Patients were identified as having severe sepsis or septic shock based on manual chart review by trained data and chart abstractors of patients with compatible International Classification of Disease-9 (ICD-9) discharge diagnosis codes as part of a broader quality improvement effort [<xref ref-type="bibr" rid="CR9">9</xref>
]. Patients were selected as candidates for chart abstraction if they were admitted directly from the ED to the intensive care unit (ICU) and either had a discharge diagnosis code related to sepsis or infection or were identified separately by quality improvement coordinators in the ICU. The ICD-9 codes were chosen to include broad representation of diagnoses associated with sepsis or infection and thus to maximize sensitivity with the knowledge that chart review would subsequently minimize potential false positive results. Encounters for all patients 14 years of age or older at the time of the encounter were included in the prospective validation.</p>
</sec>
<sec id="Sec7"><title>Methods and measurements</title>
<p>A chart review datamart was populated daily based on discharge diagnoses with subsequent chart review by trained data collectors. In addition to determining whether patients met consensus criteria [<xref ref-type="bibr" rid="CR10">10</xref>
] for severe sepsis or septic shock, data collectors also determined whether care was compliant with the Intermountain sepsis management bundles. Results of the compliance with sepsis bundles among septic patients have been reported separately [<xref ref-type="bibr" rid="CR9">9</xref>
].</p>
<p>To support improved compliance and quality of care as part of the overall bundle-driven quality improvement effort, we developed the probabilistic model whose accuracy is described in this report in both the retrospective (reported previously) and prospective models. Clinician collaborators initially identified 75 parameters that they suspected might be useful in the rapid identification of septic patients. These included basic demographic information, encounter details (e.g., time of day, day of week, method of arrival in the ED), results of laboratory tests, vital signs, other clinical measures, and the coded chief complaint. For any parameter for which multiple measures were available, the first available measurement in the ED was preferred. The components of the standard definition of the Systemic Inflammatory Response Syndrome (SIRS) [<xref ref-type="bibr" rid="CR10">10</xref>
] (e.g., heart rate greater than 90 beats per minute) were also included as candidates within the model. Clinicians also described desired performance characteristics for the model based on their <italic>a priori</italic>
beliefs about acceptable sensitivity and false positive rates.</p>
</sec>
<sec id="Sec8"><title>Alert deployment</title>
<p>Chart review from January 2006 through December 2008 was completed in January 2009. During February and March 2009, the model was reviewed for overall performance characteristics (e.g., sensitivity, false positive rate) and individual case review by clinicians to determine whether performance was satisfactory and at what threshold to trigger an alert. The decision was made to implement the sepsis alerting system and a binary threshold was selected (0.05, though known calibration issues with naïve Bayes models do not support a simple interpretation of that threshold).</p>
<p>The alert system consisted of (a) routine clinical data entry, (b) pushing the data to a research decision support environment, (c) generating a probability estimate using Netica™ (Norsys Software Corp), a tool for developing/deploying Bayesian systems using data available at 60 min from ED entry, and (d) sending a page and email to the ED charge nurse for clinical consideration and action if the threshold was met. While training of charge nurses and other clinicians was ongoing throughout the process, we did not explicitly measure the effect of training on model performance.</p>
</sec>
<sec id="Sec9"><title>Changes between model development and validation</title>
<p>There were several differences between the time periods of model development and model validation. Organizational changes included the movement of the primary center from LDS Hospital to Intermountain Medical Center midway through the development period and the implementation of the alerting system after the development period. From a system perspective, both the triage nurse identification system (described below) and the model-based alert were deployed after the development period. After initial implementation, the model-based alert was shifted from 60 min post ED admission to 90 min post ED admission in order to capture more laboratory results. Finally, the H1N1 influenza pandemic [<xref ref-type="bibr" rid="CR11">11</xref>
] occurred during the model validation period and after model development.</p>
<p>Throughout these changes, the same data processes and procedures were maintained for the source data, chart abstraction, chart review, and model assessment.</p>
</sec>
<sec id="Sec10"><title>Triage nurse identification system</title>
<p>To maximize the chances of rapid patient identification, a parallel effort was made to identify sepsis patients via a modification to the triage nurse screens in the hospital electronic medical record. This system prompted the nurse to consider whether the patient had a new infection if the patient displayed fever, tachycardia, hypotension, or tachypnea. The triage nurse information was not considered as a potential predictor for the model, as it was not available during the model development phase. Data on the triage nurse information are provided for comparison purposes only.</p>
</sec>
<sec id="Sec11"><title>Outcomes</title>
<p>The outcome of interest for model development was confirmed diagnosis of severe sepsis or septic shock. For assessment of model performance, the outcome was sensitivity > 80 % and a false positive rate < 15 alerts per day.</p>
</sec>
<sec id="Sec12"><title>Analysis</title>
<p>The methodological details of the statistical modeling environment will be described in a separate publication. Briefly, it is a naïve Bayes framework that employs forward stepwise selection based on out-of-bag area under the receiver operating characteristic curve (AUC) derived via bootstrap sampling [<xref ref-type="bibr" rid="CR12">12</xref>
, <xref ref-type="bibr" rid="CR13">13</xref>
]. The naïve Bayes model was selected due to substantial, non-random missingness (especially in laboratory values) and a desire to avoid imputation methods. Naïve Bayes deals with non-random missingness natively and performs better than traditional regression methods at non-random missingness. Briefly, a missing variable does not affect the overall probability estimate of the model. Additionally, the modeling environment allows for direct clinician input with respect to preferences for certain parameters over others—should certain parameters be of statistically equivalent utility but more easily attained, for instance. Among highly collinear variables the naïve Bayes model we employed selected the variable most predictive of outcome. Because we were interested in prediction rather than model inference, we therefore performed no analyses of collinearity. In a naïve Bayesian model, parameters do not have weights/coefficients; instead each parameter is associated with a mean and standard deviation in septic and non-septic patients, and the differences in these distributions are used to build the prediction model.</p>
<p>All model development was based on the R statistical computing environment [<xref ref-type="bibr" rid="CR14">14</xref>
].</p>
</sec>
<sec id="Sec13"><title>Model performance assessment</title>
<p>The primary success measures specified by clinicians were sensitivity (target 80 %) and the false positive count (target <15 patients per day). Based on ED visit volume, the target false positive rate equated to approximately 7.2 %. We calculated the AUC for both the continuous prediction model and the binary alert. For the binary alert, the positive predictive value (PPV), negative predictive value (NPV), and incremental (compared to the triage nurse system) true positive counts were calculated.</p>
<p>Final probability estimates from the model were calculated in Netica and then incorporated into the alerting environment; all other analyses were conducted with R.</p>
</sec>
</sec>
<sec id="Sec14"><title>Results</title>
<sec id="Sec15"><title>Characteristics of study subjects</title>
<p>From April 2009 through June 2010 (15 months) there were 93,773 ED encounters involving 58,603 unique patients at Intermountain Medical Center. Table <xref rid="Tab1" ref-type="table">1</xref>
presents patient characteristics with the encounter as the unit of analysis for those encounters not involving sepsis (<italic>n</italic>
= 93,421; 58,269 unique patients) and those involving sepsis (<italic>n</italic>
= 352 [0.38 %]; 334 unique patients). The only parameter not significantly different across groups was gender (58.2 % female for non-septic encounters and 53.7 % for septic encounters). For encounters involving sepsis, patients tended to be older (median 60.9 versus 39.6 years), had a lower blood pressure (median 109/60 versus 132/77 mmHg) within one hour of entering the ED, a higher temperature (median 37.7 versus 36.4 Celsius), a higher heart rate (median 111 versus 86 min<sup>−1</sup>
), and higher white blood count (14,300 versus 8,500 μL<sup><bold>−1</bold>
</sup>
). Measurement rates were similar for vital signs (> = 95 %) but WBC was drawn more frequently within one hour of ED admission for encounters involving sepsis (53 % versus 22 %).<table-wrap id="Tab1"><label>Table 1</label>
<caption><p>Basic encounter characteristics at 1 h from ED entry</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th rowspan="2"></th>
<th colspan="4">Non-Septic Encounters</th>
<th colspan="4">Septic Encounters</th>
<th rowspan="2"><italic>p</italic>
-value</th>
</tr>
<tr><th>Pt. Est.</th>
<th>20 %</th>
<th>75 %</th>
<th>% Avail</th>
<th>Pt. Est.</th>
<th>20 %</th>
<th>75 %</th>
<th>% Avail</th>
</tr>
</thead>
<tbody><tr><td>n</td>
<td>93,421</td>
<td colspan="3">(58,269 unique patients)</td>
<td>352</td>
<td colspan="3">(334 unique patients)</td>
<td></td>
</tr>
<tr><td>Age</td>
<td>39.6</td>
<td>27.3</td>
<td>57.1</td>
<td>100 %</td>
<td>60.9</td>
<td>51.7</td>
<td>73.4</td>
<td>100 %</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td>Female</td>
<td>58.2 %</td>
<td>57.9 %</td>
<td>58.5 %</td>
<td></td>
<td>53.7 %</td>
<td>48.3 %</td>
<td>59.0 %</td>
<td></td>
<td char="." align="char">0.0934</td>
</tr>
<tr><td>SBP</td>
<td>132</td>
<td>119</td>
<td>146</td>
<td>97 %</td>
<td>109</td>
<td>91</td>
<td>129</td>
<td>97 %</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td>DBP</td>
<td>77</td>
<td>68</td>
<td>86</td>
<td>97 %</td>
<td>60</td>
<td>48</td>
<td>73</td>
<td>97 %</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td>Mean BP</td>
<td>95</td>
<td>86</td>
<td>105</td>
<td>97 %</td>
<td>75</td>
<td>64</td>
<td>94</td>
<td>97 %</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td>Temp (C)</td>
<td>36.4</td>
<td>36.0</td>
<td>36.8</td>
<td>95 %</td>
<td>37.7</td>
<td>36.6</td>
<td>38.7</td>
<td>96 %</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td>HR</td>
<td>86</td>
<td>75</td>
<td>99</td>
<td>97 %</td>
<td>111</td>
<td>91</td>
<td>126</td>
<td>98 %</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td>WBC</td>
<td>8.5 K</td>
<td>6.7 K</td>
<td>10.9 K</td>
<td>22 %</td>
<td>14.3 K</td>
<td>9.8 K</td>
<td>20.6 K</td>
<td>53 %</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td>Lactate</td>
<td>1.7</td>
<td>1.2</td>
<td>2.5</td>
<td>3 %</td>
<td>2.8</td>
<td>1.9</td>
<td>4.7</td>
<td>36 %</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td colspan="10">ᅟ</td>
</tr>
<tr><td>Screen</td>
<td>6.8 %</td>
<td>6.6 %</td>
<td>6.9 %</td>
<td></td>
<td>61.9 %</td>
<td>56.6 %</td>
<td>67.0 %</td>
<td>0</td>
<td char="." align="char">0.0000</td>
</tr>
<tr><td>Flag</td>
<td>3.1 %</td>
<td>3.0 %</td>
<td>3.2 %</td>
<td></td>
<td>54.3 %</td>
<td>48.9 %</td>
<td>59.6 %</td>
<td>0</td>
<td char="." align="char">0.0000</td>
</tr>
</tbody>
</table>
<table-wrap-foot><p>Median and quartiles provided for continues variables</p>
<p>Mean and 95 % exact confidence intervals provided for binary variables</p>
<p><italic>SBP</italic>
Systolic blood pressure, <italic>DBP</italic>
Diastolic blood pressure, <italic>BP</italic>
blood pressure, <italic>HR</italic>
heart rate, <italic>WBC</italic>
white blood cell count</p>
</table-wrap-foot>
</table-wrap>
</p>
</sec>
<sec id="Sec16"><title>Main results</title>
<sec id="Sec17"><title>Model components</title>
<p>In the retrospective work reported previously and the prospective validation reported here, we included five parameters: temperature, heart rate, respiratory rate, white blood cell count, and mean arterial pressure.</p>
</sec>
<sec id="Sec18"><title>Overall model performance</title>
<p>Table <xref rid="Tab2" ref-type="table">2</xref>
and Fig. <xref rid="Fig1" ref-type="fig">1</xref>
present the prospective performance characteristics of the model, various alerting thresholds, and the two alternate alerting approaches (triage nurse and SIRS-derived rule) over the 15-month period. The AUC for the continuous predictor was 0.953. The pre-selected threshold for alerting was at a model generated probability of 0.05. This generated an AUC of 0.859, a sensitivity of 76.4 %, and a false positive rate of 4.7 % (median 9.64 false positive alerts per day). The triage nurse identification process generated a lower AUC (0.756), sensitivity (54.3 %), and false positive rate (3.1 %). The SIRS-derived rule generated even lower measures (AUC = 0.606, sensitivity = 21.6 %, false positive rate = 0.4 %).<table-wrap id="Tab2"><label>Table 2</label>
<caption><p>Overall performance characteristics</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th rowspan="2">Alert</th>
<th rowspan="2">Cutoff</th>
<th rowspan="2">sens</th>
<th rowspan="2">fpr</th>
<th rowspan="2">auc</th>
<th rowspan="2">spec</th>
<th rowspan="2">ppv</th>
<th rowspan="2">npv</th>
<th rowspan="2">FP/Day</th>
<th colspan="2">Inc. Sepsis TP</th>
</tr>
<tr><th>Nurse</th>
<th>Other</th>
</tr>
</thead>
<tbody><tr><td>Triangle Nurse</td>
<td char="." align="char">1.000</td>
<td char="." align="char">54.3 %</td>
<td char="." align="char">3.1 %</td>
<td char="." align="char">0.756</td>
<td char="." align="char">96.9 %</td>
<td char="." align="char">6.1 %</td>
<td char="." align="char">99.8 %</td>
<td char="." align="char">6.44</td>
<td></td>
<td></td>
</tr>
<tr><td>SIRS Derived</td>
<td char="." align="char">1.000</td>
<td char="." align="char">21.6 %</td>
<td char="." align="char">0.4 %</td>
<td char="." align="char">0.606</td>
<td char="." align="char">99.6 %</td>
<td char="." align="char">17.2 %</td>
<td char="." align="char">99.7 %</td>
<td char="." align="char">0.80</td>
<td>187</td>
<td>4</td>
</tr>
<tr><td><bold>Cut05</bold>
</td>
<td char="." align="char"><bold>0.050</bold>
</td>
<td char="." align="char"><bold>76.4 %</bold>
</td>
<td char="." align="char"><bold>4.7 %</bold>
</td>
<td char="." align="char"><bold>0.859</bold>
</td>
<td char="." align="char"><bold>95.3 %</bold>
</td>
<td char="." align="char"><bold>5.8 %</bold>
</td>
<td char="." align="char"><bold>99.9 %</bold>
</td>
<td char="." align="char"><bold>9.64</bold>
</td>
<td><bold>22</bold>
</td>
<td><bold>100</bold>
</td>
</tr>
<tr><td>Cut04</td>
<td char="." align="char">0.040</td>
<td char="." align="char">79.0 %</td>
<td char="." align="char">5.4 %</td>
<td char="." align="char">0.868</td>
<td char="." align="char">94.6 %</td>
<td char="." align="char">5.2 %</td>
<td char="." align="char">99.9 %</td>
<td char="." align="char">11.16</td>
<td>18</td>
<td>105</td>
</tr>
<tr><td>Sens80</td>
<td char="." align="char">0.036</td>
<td char="." align="char">80.1 %</td>
<td char="." align="char">5.8 %</td>
<td char="." align="char">0.871</td>
<td char="." align="char">94.2 %</td>
<td char="." align="char">4.9 %</td>
<td char="." align="char">99.9 %</td>
<td char="." align="char">12.03</td>
<td>17</td>
<td>108</td>
</tr>
<tr><td>FPC15</td>
<td char="." align="char">0.026</td>
<td char="." align="char">85.2 %</td>
<td char="." align="char">7.3 %</td>
<td char="." align="char">0.890</td>
<td char="." align="char">92.7 %</td>
<td char="." align="char">4.2 %</td>
<td char="." align="char">99.9 %</td>
<td char="." align="char">15.00</td>
<td>13</td>
<td>122</td>
</tr>
<tr><td>MaxAUC</td>
<td char="." align="char">0.017</td>
<td char="." align="char">89.2 %</td>
<td char="." align="char">9.9 %</td>
<td char="." align="char">0.897</td>
<td char="." align="char">90.1 %</td>
<td char="." align="char">3.3 %</td>
<td char="." align="char">100.0 %</td>
<td char="." align="char">20.33 %</td>
<td>12</td>
<td>135</td>
</tr>
<tr><td>NrsFP</td>
<td char="." align="char">0.097</td>
<td char="." align="char">70.2 %</td>
<td char="." align="char">3.1 %</td>
<td char="." align="char">0.835</td>
<td char="." align="char">96.9 %</td>
<td char="." align="char">7.8 %</td>
<td char="." align="char">99.9 %</td>
<td char="." align="char">6.44</td>
<td>32</td>
<td>88</td>
</tr>
<tr><td>NrsSens</td>
<td char="." align="char">0.330</td>
<td char="." align="char">54.3 %</td>
<td char="." align="char">1.6 %</td>
<td char="." align="char">0.763</td>
<td char="." align="char">98.4 %</td>
<td char="." align="char">11.4 %</td>
<td char="." align="char">99.8 %</td>
<td char="." align="char">3.27</td>
<td>64</td>
<td>64</td>
</tr>
</tbody>
</table>
<table-wrap-foot><p><italic>SIRS</italic>
systemic inflammatory response syndrome, <italic>Cut05/Cut04</italic>
Bayesian cutpoint threshold of 0.05/0.04, <italic>Sens80</italic>
threshold for sensitivity = 80 %, <italic>FPC15</italic>
threshold of 15 false positive alerts per day, <italic>MaxAUC</italic>
threshold maximizing area under the receiver operating characteristic curve, <italic>NrsFP</italic>
nurse-driven false positive threshold, <italic>NrsSens</italic>
nurse-driven sensitivity threshold. Boldface indicates the primary cutpoint used</p>
</table-wrap-foot>
</table-wrap>
<fig id="Fig1"><label>Fig. 1</label>
<caption><p>Model threshold selection based on overall performance characteristics</p>
</caption>
<graphic xlink:href="12873_2016_95_Fig1_HTML" id="MO1"></graphic>
</fig>
</p>
<p>The pre-specified goals for the alerting model were 80 % sensitivity and fewer than 15 false positives per day (false positive rate ~7.2 %). We identified different thresholds to achieve both of these targets. Selecting an alerting threshold of 0.036 would have achieved a sensitivity of 79.0 % (at goal) and false positive rate of 5.4 % (better than goal). Selecting an alerting threshold of 0.026 would have achieved a sensitivity of 85.2 % (better than goal) and false positive rate of 7.3 % (at goal). AUCs for the respective model are displayed in the Table.</p>
<p>Because the triage nurse identification process ran concurrently with the probabilistic sepsis alert model, it is possible to evaluate whether the two systems identified the same patients. The last two columns in Table <xref rid="Tab2" ref-type="table">2</xref>
indicate how many of the 352 sepsis encounters were uniquely identified only by either the triage nurse process or the sepsis alerting model. Using the pre-specified cutoff of 0.05, the triage nurse contributed an incremental 22 correct alerts, while the model contributed 100. Had the model threshold been set at 0.026 (allowing a false positive rate of 7.2 %), the triage nurse would have added 13 alerts and the model would have added 122.</p>
</sec>
<sec id="Sec19"><title>Performance by month</title>
<p>The number of ED encounters, rate of sepsis, and performance characteristics were evaluated by month (Table <xref rid="Tab3" ref-type="table">3</xref>
, Fig. <xref rid="Fig2" ref-type="fig">2</xref>
). Overall, 0.38 % of encounters involved sepsis, with a range of 0.27 to 0.47 % and a coefficient of variation (CV) of 16.72 %. The continuous predictor had an overall AUC of 0.953; with a range of 0.931 to 0.974, and a CV of 0.014 (the lowest of any measure). The sensitivity of the 0.05 threshold alert model ranged from 64 to 89 %, while its false positive rate ranged from 4.2 to 5.6 %. The triage nurse system demonstrated higher variability by month. Sensitivity ranged from 38.1 to 70.6 % and the false positive rate ranged from 2.3 to 4.1 %. In general the CV for the triage nurse process was roughly twice that of the model-based alert.<table-wrap id="Tab3"><label>Table 3</label>
<caption><p>Monthly performance characteristics</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th>Total:</th>
<th>93,773</th>
<th>208</th>
<th>352</th>
<th>0.38 %</th>
<th rowspan="2">Cont. AUC</th>
<th colspan="4">Model Cut > =0.05</th>
<th colspan="4">Triage Nurse</th>
</tr>
<tr><th>Month</th>
<th>Encs</th>
<th>Per Day</th>
<th>Sepsis</th>
<th>Rate</th>
<th>Sens</th>
<th>FPR</th>
<th>FP/Day</th>
<th>AUC</th>
<th>Sens</th>
<th>FPR</th>
<th>FP/Day</th>
<th>AUC</th>
</tr>
</thead>
<tbody><tr><td>200904</td>
<td char="," align="char">6,279</td>
<td char="." align="char">209</td>
<td char="." align="char">22</td>
<td char="." align="char">0.35 %</td>
<td char="." align="char">0.957</td>
<td char="." align="char">72.7 %</td>
<td char="." align="char">4.5 %</td>
<td char="." align="char">9.43</td>
<td char="." align="char">0.841</td>
<td char="." align="char">68.2 %</td>
<td char="." align="char">2.7 %</td>
<td char="." align="char">5.69</td>
<td char="." align="char">0.827</td>
</tr>
<tr><td>200905</td>
<td char="," align="char">6,583</td>
<td char="." align="char">212</td>
<td char="." align="char">21</td>
<td char="." align="char">0.32 %</td>
<td char="." align="char">0.952</td>
<td char="." align="char">76.2 %</td>
<td char="." align="char">5.2 %</td>
<td char="." align="char">11.00</td>
<td char="." align="char">0.855</td>
<td char="." align="char">38.1 %</td>
<td char="." align="char">2.8 %</td>
<td char="." align="char">5.86</td>
<td char="." align="char">0.677</td>
</tr>
<tr><td>200906</td>
<td char="," align="char">6,860</td>
<td char="." align="char">229</td>
<td char="." align="char">26</td>
<td char="." align="char">0.38 %</td>
<td char="." align="char">0.963</td>
<td char="." align="char">88.5 %</td>
<td char="." align="char">5.6 %</td>
<td char="." align="char">12.82</td>
<td char="." align="char">0.914</td>
<td char="." align="char">50.0 %</td>
<td char="." align="char">4.1 %</td>
<td char="." align="char">9.27</td>
<td char="." align="char">0.730</td>
</tr>
<tr><td>200907</td>
<td char="," align="char">6,736</td>
<td char="." align="char">217</td>
<td char="." align="char">22</td>
<td char="." align="char">0.33 %</td>
<td char="." align="char">0.974</td>
<td char="." align="char">72.7 %</td>
<td char="." align="char">4.6 %</td>
<td char="." align="char">10.03</td>
<td char="." align="char">0.841</td>
<td char="." align="char">50.0 %</td>
<td char="." align="char">2.3 %</td>
<td char="." align="char">4.92</td>
<td char="." align="char">0.739</td>
</tr>
<tr><td>200908</td>
<td char="," align="char">6,439</td>
<td char="." align="char">208</td>
<td char="." align="char">22</td>
<td char="." align="char">0.34 %</td>
<td char="." align="char">0.934</td>
<td char="." align="char">72.7 %</td>
<td char="." align="char">4.3 %</td>
<td char="." align="char">8.97</td>
<td char="." align="char">0.842</td>
<td char="." align="char">45.5 %</td>
<td char="." align="char">2.5 %</td>
<td char="." align="char">5.18</td>
<td char="." align="char">0.715</td>
</tr>
<tr><td>200909</td>
<td char="," align="char">6,352</td>
<td char="." align="char">212</td>
<td char="." align="char">25</td>
<td char="." align="char">0.39 %</td>
<td char="." align="char">0.961</td>
<td char="." align="char">64.0 %</td>
<td char="." align="char">4.6 %</td>
<td char="." align="char">9.64</td>
<td char="." align="char">0.797</td>
<td char="." align="char">40.0 %</td>
<td char="." align="char">2.6 %</td>
<td char="." align="char">5.59</td>
<td char="." align="char">0.687</td>
</tr>
<tr><td>200910</td>
<td char="," align="char">6,451</td>
<td char="." align="char">208</td>
<td char="." align="char">29</td>
<td char="." align="char">0.45 %</td>
<td char="." align="char">0.957</td>
<td char="." align="char">86.2 %</td>
<td char="." align="char">4.5 %</td>
<td char="." align="char">9.43</td>
<td char="." align="char">0.908</td>
<td char="." align="char">69.0 %</td>
<td char="." align="char">3.6 %</td>
<td char="." align="char">7.49</td>
<td char="." align="char">0.627</td>
</tr>
<tr><td>200911</td>
<td char="," align="char">5,975</td>
<td char="." align="char">199</td>
<td char="." align="char">25</td>
<td char="." align="char">0.42 %</td>
<td char="." align="char">0.953</td>
<td char="." align="char">76.0 %</td>
<td char="." align="char">4.5 %</td>
<td char="." align="char">8.94</td>
<td char="." align="char">0.858</td>
<td char="." align="char">56.0 %</td>
<td char="." align="char">3.5 %</td>
<td char="." align="char">7.00</td>
<td char="." align="char">0.762</td>
</tr>
<tr><td>200912</td>
<td char="," align="char">5,753</td>
<td char="." align="char">186</td>
<td char="." align="char">27</td>
<td char="." align="char">0.47 %</td>
<td char="." align="char">0.944</td>
<td char="." align="char">81.5 %</td>
<td char="." align="char">4.7 %</td>
<td char="." align="char">8.75</td>
<td char="." align="char">0.884</td>
<td char="." align="char">51.9 %</td>
<td char="." align="char">3.0 %</td>
<td char="." align="char">5.51</td>
<td char="." align="char">0.744</td>
</tr>
<tr><td>201001</td>
<td char="," align="char">6,206</td>
<td char="." align="char">200</td>
<td char="." align="char">27</td>
<td char="." align="char">0.44 %</td>
<td char="." align="char">0.932</td>
<td char="." align="char">77.8 %</td>
<td char="." align="char">4.2 %</td>
<td char="." align="char">8.49</td>
<td char="." align="char">0.868</td>
<td char="." align="char">44.4 %</td>
<td char="." align="char">3.4 %</td>
<td char="." align="char">6.71</td>
<td char="." align="char">0.705</td>
</tr>
<tr><td>201002</td>
<td char="," align="char">5,568</td>
<td char="." align="char">199</td>
<td char="." align="char">22</td>
<td char="." align="char">0.40 %</td>
<td char="." align="char">0.948</td>
<td char="." align="char">68.2 %</td>
<td char="." align="char">5.1 %</td>
<td char="." align="char">10.11</td>
<td char="." align="char">0.815</td>
<td char="." align="char">63.6 %</td>
<td char="." align="char">3.5 %</td>
<td char="." align="char">6.96</td>
<td char="." align="char">0.801</td>
</tr>
<tr><td>201003</td>
<td char="," align="char">6,250</td>
<td char="." align="char">202</td>
<td char="." align="char">17</td>
<td char="." align="char">0.27 %</td>
<td char="." align="char">0.965</td>
<td char="." align="char">82.4 %</td>
<td char="." align="char">4.6 %</td>
<td char="." align="char">9.28</td>
<td char="." align="char">0.889</td>
<td char="." align="char">70.6 %</td>
<td char="." align="char">3.8 %</td>
<td char="." align="char">7.63</td>
<td char="." align="char">0.834</td>
</tr>
<tr><td>201004</td>
<td char="," align="char">5,885</td>
<td char="." align="char">196</td>
<td char="." align="char">27</td>
<td char="." align="char">0.46 %</td>
<td char="." align="char">0.947</td>
<td char="." align="char">77.8 %</td>
<td char="." align="char">4.4 %</td>
<td char="." align="char">8.71</td>
<td char="." align="char">0.867</td>
<td char="." align="char">55.6 %</td>
<td char="." align="char">2.9 %</td>
<td char="." align="char">5.73</td>
<td char="." align="char">0.763</td>
</tr>
<tr><td>201005</td>
<td char="," align="char">6,184</td>
<td char="." align="char">199</td>
<td char="." align="char">17</td>
<td char="." align="char">0.27 %</td>
<td char="." align="char">0.969</td>
<td char="." align="char">76.5 %</td>
<td char="." align="char">4.9 %</td>
<td char="." align="char">9.80</td>
<td char="." align="char">0.858</td>
<td char="." align="char">58.8 %</td>
<td char="." align="char">3.4 %</td>
<td char="." align="char">6.79</td>
<td char="." align="char">0.777</td>
</tr>
<tr><td>201006</td>
<td char="," align="char">6,252</td>
<td char="." align="char">208</td>
<td char="." align="char">23</td>
<td char="." align="char">0.37 %</td>
<td char="." align="char">0.931</td>
<td char="." align="char">69.6 %</td>
<td char="." align="char">4.4 %</td>
<td char="." align="char">9.10</td>
<td char="." align="char">0.826</td>
<td char="." align="char">56.5 %</td>
<td char="." align="char">3.0 %</td>
<td char="." align="char">6.19</td>
<td char="." align="char">0.768</td>
</tr>
<tr><td colspan="3"></td>
<td char="." align="char">Min:</td>
<td char="." align="char">0.27 %</td>
<td char="." align="char">0.931</td>
<td char="." align="char">64.0 %</td>
<td char="." align="char">4.2 %</td>
<td char="." align="char">8.49</td>
<td char="." align="char">0.797</td>
<td char="." align="char">38.1 %</td>
<td char="." align="char">2.3 %</td>
<td char="." align="char">4.92</td>
<td char="." align="char">0.677</td>
</tr>
<tr><td colspan="3"></td>
<td char="." align="char">Max:</td>
<td char="." align="char">0.47 %</td>
<td char="." align="char">0.974</td>
<td char="." align="char">88.5 %</td>
<td char="." align="char">5.6 %</td>
<td char="." align="char">12.82</td>
<td char="." align="char">0.914</td>
<td char="." align="char">70.6 %</td>
<td char="." align="char">4.1 %</td>
<td char="." align="char">9.27</td>
<td char="." align="char">0.834</td>
</tr>
<tr><td colspan="3"></td>
<td char="." align="char">CV:</td>
<td char="." align="char">16.72 %</td>
<td char="." align="char">0.014</td>
<td char="." align="char">8.7 %</td>
<td char="." align="char">7.9 %</td>
<td char="." align="char">0.11</td>
<td char="." align="char">0.038</td>
<td char="." align="char">18.8 %</td>
<td char="." align="char">16.4 %</td>
<td char="." align="char">0.18</td>
<td char="." align="char">0.066</td>
</tr>
</tbody>
</table>
</table-wrap>
<fig id="Fig2"><label>Fig. 2</label>
<caption><p>Monthly area under the curve for sepsis alert model and triage nurse</p>
</caption>
<graphic xlink:href="12873_2016_95_Fig2_HTML" id="MO2"></graphic>
</fig>
</p>
</sec>
<sec id="Sec20"><title>Practical observations during model development</title>
<p>During the development phase we found that clinicians tended to reject alerts based solely on vital signs (i.e., lacking laboratory indicators). Many clinicians expected serum lactate to be a useful distinguisher, but this was not the case because serum lactate was available for <5 % of non-septic patients within a one-hour timeframe. White blood cell count (WBC) was found to be useful in the model development phase, though it was only available approximately 20 % of the time. Due to the reluctance of clinicians to accept alerts without elevated WBC, the decision was made to delay the alert from 1 to 1.5 h post ED entry. Technical performance did not improve appreciably with the modification, but communication with clinicians (and trust in the alert) improved substantially.</p>
<p>The primary limitation of this study is the fact that validation was conducted at a single center. Mitigating this limitation is the fact that many of the cases used to develop the model came from a different hospital from the one used for the prospective validation. Additionally, there were enough contextual differences between model development and validation that we are less concerned about the generalizability of the results. There are some limitations that may constrain the export of this sepsis alert model to other institutions. First, our system requires electronic charting of ED triage, vital signs, and laboratory test results. Second, measurement of the performance characteristics of this model requires chart review to confirm the presence of sepsis. This limitation could be obviated by only requiring periodic tests of calibration. We also acknowledge that our case-finding strategy was limited to cases of severe sepsis and septic shock that required ICU admission. While we felt it important to focus on those higher-risk patients, the model will require independent validation among patients with severe sepsis who do not require ICU admission.</p>
</sec>
</sec>
</sec>
<sec id="Sec21"><title>Discussion</title>
<p>Though early identification has been suggested as a key step in treating sepsis and decreasing sepsis-associated mortality, there is little information on how to efficiently and accurately identify septic patients in a usual care setting consistent with typical workflows. The results reported here demonstrate that it was possible to build a probabilistic model to identify septic patients and select an alerting threshold to achieve the pre-specified performance goals of 80 % sensitivity and a ~7.2 % false positive rate. The selected alerting threshold for the model was close to but missed the target sensitivity while performing better than the false positive target.</p>
<p>We also intended this study to demonstrate the process by which such an alert could be deployed. One of the biggest issues facing any alert system is “alert fatigue” [<xref ref-type="bibr" rid="CR15">15</xref>
]. The decision to deploy this particular sepsis alert model came in the context of a failed prior attempt in which performance of a rule-based alerting tool was over-estimated and an initial roll-out of the triage nurse process that generated too many false positives and lead to the disengagement of clinicians. Our decision to use the 0.05 cutoff, intended to avoid alert fatigue, meant that observed sensitivity was just short (76 %) of the pre-specified sensitivity threshold (80 %).</p>
<p>While aided by organizational factors such as the concurrent quality improvement effort, we believe that the implementation of this sepsis alerting system was facilitated by clear performance target setting and individual case review. In particular, the targets were set in terms meaningful to clinicians: 8 out of 10 sepsis patients identified and <15 false alerts per day. The latter is particularly dependent upon local circumstances (e.g., propensity to alert fatigue, ED load, staffing, institutional culture) in addition to the obvious dependence on institutional volumes. While the AUC commonly used to assess the discrimination of predictive models suggested that our model performed well, we believe that the sensitivity and false positive rate are more relevant indicators for ED clinicians.</p>
</sec>
<sec id="Sec22"><title>Conclusions</title>
<p>We have demonstrated that a probabilistic sepsis alert model can be deployed in a large hospital ED to identify sepsis patients according to the goals of 80 % sensitivity and a false positive rate of ~7.2 %. Additionally, it appears that the developed model behaved as expected despite significant contextual changes between the model development and validation periods. Though measures such as the positive predictive value and incremental patients identified by the described triage nurse process recapitulate the need for engaged clinicians in any form, we believe this alert approach can offer an opportunity for solid, stable augmentation of diagnostic processes relevant to the early detection of sepsis in the ED. Finally, the ability to tune alerting thresholds may facilitate balancing local goals and tolerances for error.</p>
</sec>
</body>
<back><glossary><title>Abbreviations</title>
<def-list><def-item><term>AUC</term>
<def><p>Area under the receiver operating characteristic curve</p>
</def>
</def-item>
<def-item><term>CV</term>
<def><p>Coefficient of variability</p>
</def>
</def-item>
<def-item><term>ED</term>
<def><p>Emergency department</p>
</def>
</def-item>
<def-item><term>ICD-9</term>
<def><p>International classification of diseases-9</p>
</def>
</def-item>
<def-item><term>ICU</term>
<def><p>Intensive care unit</p>
</def>
</def-item>
<def-item><term>NPV</term>
<def><p>Negative predictive value</p>
</def>
</def-item>
<def-item><term>PPV</term>
<def><p>Positive predictive value</p>
</def>
</def-item>
<def-item><term>SIRS</term>
<def><p>Systemic inflammatory response syndrome</p>
</def>
</def-item>
</def-list>
</glossary>
<ack><p>Not applicable.</p>
<sec id="FPar1"><title>Funding</title>
<p>Dr. Brown is supported by NIGMS (K23GM094465).</p>
</sec>
<sec id="FPar2"><title>Availability of data and materials</title>
<p>In order to protect patient privacy and comply with relevant regulations, identified data are unavailable. Requests for deidentified data from qualified researchers with appropriate ethics board approvals and relevant data use agreements will be processed by the Intermountain Office of Research, officeofresearch@imail.org.</p>
</sec>
<sec id="FPar3"><title>Authors’ contributions</title>
<p>JJ, KK, and PH conceived the study. JJ, SB, TA, PH designed the study. JJ, KK, RK, TA, and PH supervised the conduct of the study and data collection. JJ, KK, and RK managed the data, including quality control. JJ and SB provided statistical advice on study design and analyzed the data. SB and JJ drafted the manuscript, and all authors contributed substantially to its revision. JJ takes responsibility for the paper as a whole. All authors read and approved the final manuscript.</p>
</sec>
<sec id="FPar4"><title>Competing interests</title>
<p>The authors declare that they have no competing interests.</p>
</sec>
<sec id="FPar5"><title>Consent for publication</title>
<p>Not applicable.</p>
</sec>
<sec id="FPar6"><title>Ethics approval and consent to participate</title>
<p>This project was approved by the Intermountain Privacy Board, the body which approves publication of results of quality improvement efforts within Intermountain Healthcare. Such projects do not require separate Institutional Review Board oversight.</p>
</sec>
</ack>
<ref-list id="Bib1"><title>References</title>
<ref id="CR1"><label>1.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Angus</surname>
<given-names>DC</given-names>
</name>
<name><surname>Linde-Zwirble</surname>
<given-names>WT</given-names>
</name>
<name><surname>Lidicker</surname>
<given-names>J</given-names>
</name>
<name><surname>Clermont</surname>
<given-names>G</given-names>
</name>
<name><surname>Carcillo</surname>
<given-names>J</given-names>
</name>
<name><surname>Pinsky</surname>
<given-names>MR</given-names>
</name>
</person-group>
<article-title>Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care</article-title>
<source>Crit Care Med</source>
<year>2001</year>
<volume>29</volume>
<fpage>1303</fpage>
<lpage>1310</lpage>
<pub-id pub-id-type="doi">10.1097/00003246-200107000-00002</pub-id>
<pub-id pub-id-type="pmid">11445675</pub-id>
</element-citation>
</ref>
<ref id="CR2"><label>2.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Martin</surname>
<given-names>GS</given-names>
</name>
<name><surname>Mannino</surname>
<given-names>DM</given-names>
</name>
<name><surname>Eaton</surname>
<given-names>S</given-names>
</name>
<name><surname>Moss</surname>
<given-names>M</given-names>
</name>
</person-group>
<article-title>The epidemiology of sepsis in the United States from 1979 through 2000</article-title>
<source>N Engl J Med</source>
<year>2003</year>
<volume>348</volume>
<fpage>1546</fpage>
<lpage>1554</lpage>
<pub-id pub-id-type="doi">10.1056/NEJMoa022139</pub-id>
<pub-id pub-id-type="pmid">12700374</pub-id>
</element-citation>
</ref>
<ref id="CR3"><label>3.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rivers</surname>
<given-names>E</given-names>
</name>
<name><surname>Nguyen</surname>
<given-names>B</given-names>
</name>
<name><surname>Havstad</surname>
<given-names>S</given-names>
</name>
<name><surname>Ressler</surname>
<given-names>J</given-names>
</name>
<name><surname>Muzzin</surname>
<given-names>A</given-names>
</name>
<name><surname>Knoblich</surname>
<given-names>B</given-names>
</name>
<name><surname>Peterson</surname>
<given-names>E</given-names>
</name>
<name><surname>Tomlanovich</surname>
<given-names>M</given-names>
</name>
</person-group>
<article-title>Early goal-directed therapy in the treatment of severe sepsis and septic shock</article-title>
<source>N Engl J Med</source>
<year>2001</year>
<volume>345</volume>
<fpage>1368</fpage>
<lpage>1377</lpage>
<pub-id pub-id-type="doi">10.1056/NEJMoa010307</pub-id>
<pub-id pub-id-type="pmid">11794169</pub-id>
</element-citation>
</ref>
<ref id="CR4"><label>4.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><collab>ProCess Investigators</collab>
<name><surname>Yealy</surname>
<given-names>DM</given-names>
</name>
<name><surname>Kellum</surname>
<given-names>JA</given-names>
</name>
<name><surname>Huang</surname>
<given-names>DT</given-names>
</name>
<name><surname>Barnato</surname>
<given-names>AE</given-names>
</name>
<name><surname>Weissfeld</surname>
<given-names>LA</given-names>
</name>
<name><surname>Pike</surname>
<given-names>F</given-names>
</name>
<name><surname>Terndrup</surname>
<given-names>T</given-names>
</name>
<name><surname>Wang</surname>
<given-names>HE</given-names>
</name>
<name><surname>Hou</surname>
<given-names>PC</given-names>
</name>
<etal></etal>
</person-group>
<article-title>A randomized trial of protocol-based care for early septic shock</article-title>
<source>N Engl J Med</source>
<year>2014</year>
<volume>370</volume>
<fpage>1683</fpage>
<lpage>1693</lpage>
<pub-id pub-id-type="doi">10.1056/NEJMoa1401602</pub-id>
<pub-id pub-id-type="pmid">24635773</pub-id>
</element-citation>
</ref>
<ref id="CR5"><label>5.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Lilly</surname>
<given-names>CM</given-names>
</name>
</person-group>
<article-title>The ProCESS trial--a new era of sepsis management</article-title>
<source>N Engl J Med</source>
<year>2014</year>
<volume>370</volume>
<fpage>1750</fpage>
<lpage>1751</lpage>
<pub-id pub-id-type="doi">10.1056/NEJMe1402564</pub-id>
<pub-id pub-id-type="pmid">24635774</pub-id>
</element-citation>
</ref>
<ref id="CR6"><label>6.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dellinger</surname>
<given-names>RP</given-names>
</name>
<name><surname>Levy</surname>
<given-names>MM</given-names>
</name>
<name><surname>Rhodes</surname>
<given-names>A</given-names>
</name>
<name><surname>Annane</surname>
<given-names>D</given-names>
</name>
<name><surname>Gerlach</surname>
<given-names>H</given-names>
</name>
<name><surname>Opal</surname>
<given-names>SM</given-names>
</name>
<name><surname>Sevransky</surname>
<given-names>JE</given-names>
</name>
<name><surname>Sprung</surname>
<given-names>CL</given-names>
</name>
<name><surname>Douglas</surname>
<given-names>IS</given-names>
</name>
<name><surname>Jaeschke</surname>
<given-names>R</given-names>
</name>
<etal></etal>
</person-group>
<article-title>Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012</article-title>
<source>Crit Care Med</source>
<year>2013</year>
<volume>41</volume>
<fpage>580</fpage>
<lpage>637</lpage>
<pub-id pub-id-type="doi">10.1097/CCM.0b013e31827e83af</pub-id>
<pub-id pub-id-type="pmid">23353941</pub-id>
</element-citation>
</ref>
<ref id="CR7"><label>7.</label>
<mixed-citation publication-type="other">Miller RR, 3rd, Dong L, Nelson NC, Brown SM, Kuttler KG, Probst DR, Allen TL, Clemmer TP, for the Intermountain Healthcare Intensive Medicine Clinical P. Multicenter Implementation of a Severe Sepsis and Septic Shock Treatment Bundle. Am J Respir Crit Care Med. 2013;188(1):77–82.</mixed-citation>
</ref>
<ref id="CR8"><label>8.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Jones</surname>
<given-names>JP</given-names>
</name>
<name><surname>Kuttler</surname>
<given-names>K</given-names>
</name>
<name><surname>Keddington</surname>
<given-names>RK</given-names>
</name>
<name><surname>Allen</surname>
<given-names>TL</given-names>
</name>
<name><surname>Brown</surname>
<given-names>SM</given-names>
</name>
<name><surname>Haug</surname>
<given-names>PJ</given-names>
</name>
</person-group>
<article-title>Derivation and validation of a probabilistic model deployed in the emergency department to identify patients with severe sepsis or septic shock</article-title>
<source>Am J Respir Crit Care Med</source>
<year>2012</year>
<volume>185</volume>
<fpage>A1126</fpage>
</element-citation>
</ref>
<ref id="CR9"><label>9.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Miller</surname>
<given-names>RR</given-names>
<suffix>3rd</suffix>
</name>
<name><surname>Dong</surname>
<given-names>L</given-names>
</name>
<name><surname>Nelson</surname>
<given-names>NC</given-names>
</name>
<name><surname>Brown</surname>
<given-names>SM</given-names>
</name>
<name><surname>Kuttler</surname>
<given-names>KG</given-names>
</name>
<name><surname>Probst</surname>
<given-names>DR</given-names>
</name>
<name><surname>Allen</surname>
<given-names>TL</given-names>
</name>
<name><surname>Clemmer</surname>
<given-names>TP</given-names>
</name>
<collab>Intermountain Healthcare Intensive Medicine Clinical P</collab>
</person-group>
<article-title>Multicenter implementation of a severe sepsis and septic shock treatment bundle</article-title>
<source>Am J Respir Crit Care Med</source>
<year>2013</year>
<volume>188</volume>
<fpage>77</fpage>
<lpage>82</lpage>
<pub-id pub-id-type="doi">10.1164/rccm.201212-2199OC</pub-id>
<pub-id pub-id-type="pmid">23631750</pub-id>
</element-citation>
</ref>
<ref id="CR10"><label>10.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bone</surname>
<given-names>RC</given-names>
</name>
<name><surname>Balk</surname>
<given-names>RA</given-names>
</name>
<name><surname>Cerra</surname>
<given-names>FB</given-names>
</name>
<name><surname>Dellinger</surname>
<given-names>RP</given-names>
</name>
<name><surname>Fein</surname>
<given-names>AM</given-names>
</name>
<name><surname>Knaus</surname>
<given-names>WA</given-names>
</name>
<name><surname>Schein</surname>
<given-names>RM</given-names>
</name>
<name><surname>Sibbald</surname>
<given-names>WJ</given-names>
</name>
</person-group>
<article-title>Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM consensus conference committee. American college of chest physicians/society of critical care medicine</article-title>
<source>Chest</source>
<year>1992</year>
<volume>101</volume>
<fpage>1644</fpage>
<lpage>1655</lpage>
<pub-id pub-id-type="doi">10.1378/chest.101.6.1644</pub-id>
<pub-id pub-id-type="pmid">1303622</pub-id>
</element-citation>
</ref>
<ref id="CR11"><label>11.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Miller</surname>
<given-names>RR</given-names>
<suffix>3rd</suffix>
</name>
<name><surname>Markewitz</surname>
<given-names>BA</given-names>
</name>
<name><surname>Rolfs</surname>
<given-names>RT</given-names>
</name>
<name><surname>Brown</surname>
<given-names>SM</given-names>
</name>
<name><surname>Dascomb</surname>
<given-names>KK</given-names>
</name>
<name><surname>Grissom</surname>
<given-names>CK</given-names>
</name>
<name><surname>Friedrichs</surname>
<given-names>MD</given-names>
</name>
<name><surname>Mayer</surname>
<given-names>J</given-names>
</name>
<name><surname>Hirshberg</surname>
<given-names>EL</given-names>
</name>
<name><surname>Conklin</surname>
<given-names>J</given-names>
</name>
<etal></etal>
</person-group>
<article-title>Clinical findings and demographic factors associated with ICU admission in Utah due to novel 2009 influenza A(H1N1) infection</article-title>
<source>Chest</source>
<year>2010</year>
<volume>137</volume>
<fpage>752</fpage>
<lpage>758</lpage>
<pub-id pub-id-type="doi">10.1378/chest.09-2517</pub-id>
<pub-id pub-id-type="pmid">19933372</pub-id>
</element-citation>
</ref>
<ref id="CR12"><label>12.</label>
<element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Harrell</surname>
<given-names>FE</given-names>
</name>
</person-group>
<source>Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis</source>
<year>2001</year>
<publisher-loc>New York</publisher-loc>
<publisher-name>Springer</publisher-name>
</element-citation>
</ref>
<ref id="CR13"><label>13.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Efron</surname>
<given-names>B</given-names>
</name>
<name><surname>Tibshirani</surname>
<given-names>R</given-names>
</name>
</person-group>
<article-title>Improvements on cross-validation: The.632+ bootstrap method</article-title>
<source>J Am Stat Assoc</source>
<year>1997</year>
<volume>92</volume>
<fpage>548</fpage>
<lpage>560</lpage>
</element-citation>
</ref>
<ref id="CR14"><label>14.</label>
<element-citation publication-type="book"><person-group person-group-type="author"><collab>R Development Core Team</collab>
</person-group>
<source>R: A Language and Environment for Statistical Computing</source>
<year>2009</year>
<publisher-loc>R Foundation for Statistical Computing</publisher-loc>
<publisher-name>Vienna, Austria</publisher-name>
</element-citation>
</ref>
<ref id="CR15"><label>15.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>van der Sijs</surname>
<given-names>H</given-names>
</name>
<name><surname>Aarts</surname>
<given-names>J</given-names>
</name>
<name><surname>Vulto</surname>
<given-names>A</given-names>
</name>
<name><surname>Berg</surname>
<given-names>M</given-names>
</name>
</person-group>
<article-title>Overriding of drug safety alerts in computerized physician order entry</article-title>
<source>J Am Med Inform Assoc</source>
<year>2006</year>
<volume>13</volume>
<fpage>138</fpage>
<lpage>147</lpage>
<pub-id pub-id-type="doi">10.1197/jamia.M1809</pub-id>
<pub-id pub-id-type="pmid">16357358</pub-id>
</element-citation>
</ref>
</ref-list>
</back>
</pmc>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Sante/explor/PandemieGrippaleV1/Data/Pmc/Corpus
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000169 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd -nk 000169 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Sante |area= PandemieGrippaleV1 |flux= Pmc |étape= Corpus |type= RBID |clé= PMC:4994262 |texte= Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/RBID.i -Sk "pubmed:27549755" \ | HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd \ | NlmPubMed2Wicri -a PandemieGrippaleV1
![]() | This area was generated with Dilib version V0.6.34. | ![]() |