New models and online calculator for predicting non-sentinel lymph node status in sentinel lymph node positive breast cancer patients
Identifieur interne : 001B05 ( Pmc/Corpus ); précédent : 001B04; suivant : 001B06New models and online calculator for predicting non-sentinel lymph node status in sentinel lymph node positive breast cancer patients
Auteurs : Holbrook E. Kohrt ; Richard A. Olshen ; Honnie R. Bermas ; William H. Goodson ; Douglas J. Wood ; Solomon Henry ; Robert V. Rouse ; Lisa Bailey ; Vicki J. Philben ; Frederick M. Dirbas ; Jocelyn J. Dunn ; Denise L. Johnson ; Irene L. Wapnir ; Robert W. Carlson ; Frank E. Stockdale ; Nora M. Hansen ; Stefanie S. JeffreySource :
- BMC Cancer [ 1471-2407 ] ; 2008.
Abstract
Current practice is to perform a completion axillary lymph node dissection (ALND) for breast cancer patients with tumor-involved sentinel lymph nodes (SLNs), although fewer than half will have non-sentinel node (NSLN) metastasis. Our goal was to develop new models to quantify the risk of NSLN metastasis in SLN-positive patients and to compare predictive capabilities to another widely used model.
We constructed three models to predict NSLN status: recursive partitioning with receiver operating characteristic curves (RP-ROC), boosted Classification and Regression Trees (CART), and multivariate logistic regression (MLR) informed by CART. Data were compiled from a multicenter Northern California and Oregon database of 784 patients who prospectively underwent SLN biopsy and completion ALND. We compared the predictive abilities of our best model and the Memorial Sloan-Kettering Breast Cancer Nomogram (Nomogram) in our dataset and an independent dataset from Northwestern University.
285 patients had positive SLNs, of which 213 had known angiolymphatic invasion status and 171 had complete pathologic data including hormone receptor status. 264 (93%) patients had limited SLN disease (micrometastasis, 70%, or isolated tumor cells, 23%). 101 (35%) of all SLN-positive patients had tumor-involved NSLNs. Three variables (tumor size, angiolymphatic invasion, and SLN metastasis size) predicted risk in all our models. RP-ROC and boosted CART stratified patients into four risk levels. MLR informed by CART was most accurate. Using two composite predictors calculated from three variables, MLR informed by CART was more accurate than the Nomogram computed using eight predictors. In our dataset, area under ROC curve (AUC) was 0.83/0.85 for MLR (n = 213/n = 171) and 0.77 for Nomogram (n = 171). When applied to an independent dataset (n = 77), AUC was 0.74 for our model and 0.62 for Nomogram. The composite predictors in our model were the product of angiolymphatic invasion and size of SLN metastasis, and the product of tumor size and square of SLN metastasis size.
We present a new model developed from a community-based SLN database that uses only three rather than eight variables to achieve higher accuracy than the Nomogram for predicting NSLN status in two different datasets.
Url:
DOI: 10.1186/1471-2407-8-66
PubMed: 18315887
PubMed Central: 2311316
Links to Exploration step
PMC:2311316Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">New models and online calculator for predicting non-sentinel lymph node status in sentinel lymph node positive breast cancer patients</title><author><name sortKey="Kohrt, Holbrook E" sort="Kohrt, Holbrook E" uniqKey="Kohrt H" first="Holbrook E" last="Kohrt">Holbrook E. Kohrt</name><affiliation><nlm:aff id="I1">Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Olshen, Richard A" sort="Olshen, Richard A" uniqKey="Olshen R" first="Richard A" last="Olshen">Richard A. Olshen</name><affiliation><nlm:aff id="I2">Department of Health Research and Policy-Biostatistics, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation><affiliation><nlm:aff id="I3">Departments of Statistics and Electrical Engineering, Stanford University, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Bermas, Honnie R" sort="Bermas, Honnie R" uniqKey="Bermas H" first="Honnie R" last="Bermas">Honnie R. Bermas</name><affiliation><nlm:aff id="I4">Department of Surgery, Northwestern University Feinberg School of Medicine and Lynn Sage Comprehensive Breast Center, Northwestern Memorial Hospital, Chicago, IL, USA</nlm:aff></affiliation></author><author><name sortKey="Goodson, William H" sort="Goodson, William H" uniqKey="Goodson W" first="William H" last="Goodson">William H. Goodson</name><affiliation><nlm:aff id="I5">Department of Surgery, California Pacific Medical Center, San Francisco, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Wood, Douglas J" sort="Wood, Douglas J" uniqKey="Wood D" first="Douglas J" last="Wood">Douglas J. Wood</name><affiliation><nlm:aff id="I2">Department of Health Research and Policy-Biostatistics, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Henry, Solomon" sort="Henry, Solomon" uniqKey="Henry S" first="Solomon" last="Henry">Solomon Henry</name><affiliation><nlm:aff id="I2">Department of Health Research and Policy-Biostatistics, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Rouse, Robert V" sort="Rouse, Robert V" uniqKey="Rouse R" first="Robert V" last="Rouse">Robert V. Rouse</name><affiliation><nlm:aff id="I6">Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Bailey, Lisa" sort="Bailey, Lisa" uniqKey="Bailey L" first="Lisa" last="Bailey">Lisa Bailey</name><affiliation><nlm:aff id="I7">Department of Surgery, Alta Bates Summit Medical Center, Berkeley, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Philben, Vicki J" sort="Philben, Vicki J" uniqKey="Philben V" first="Vicki J" last="Philben">Vicki J. Philben</name><affiliation><nlm:aff id="I8">Department of Surgery, Mercy Medical Center, Redding, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Dirbas, Frederick M" sort="Dirbas, Frederick M" uniqKey="Dirbas F" first="Frederick M" last="Dirbas">Frederick M. Dirbas</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Dunn, Jocelyn J" sort="Dunn, Jocelyn J" uniqKey="Dunn J" first="Jocelyn J" last="Dunn">Jocelyn J. Dunn</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Johnson, Denise L" sort="Johnson, Denise L" uniqKey="Johnson D" first="Denise L" last="Johnson">Denise L. Johnson</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Wapnir, Irene L" sort="Wapnir, Irene L" uniqKey="Wapnir I" first="Irene L" last="Wapnir">Irene L. Wapnir</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Carlson, Robert W" sort="Carlson, Robert W" uniqKey="Carlson R" first="Robert W" last="Carlson">Robert W. Carlson</name><affiliation><nlm:aff id="I1">Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Stockdale, Frank E" sort="Stockdale, Frank E" uniqKey="Stockdale F" first="Frank E" last="Stockdale">Frank E. Stockdale</name><affiliation><nlm:aff id="I1">Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Hansen, Nora M" sort="Hansen, Nora M" uniqKey="Hansen N" first="Nora M" last="Hansen">Nora M. Hansen</name><affiliation><nlm:aff id="I4">Department of Surgery, Northwestern University Feinberg School of Medicine and Lynn Sage Comprehensive Breast Center, Northwestern Memorial Hospital, Chicago, IL, USA</nlm:aff></affiliation></author><author><name sortKey="Jeffrey, Stefanie S" sort="Jeffrey, Stefanie S" uniqKey="Jeffrey S" first="Stefanie S" last="Jeffrey">Stefanie S. Jeffrey</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">PMC</idno><idno type="pmid">18315887</idno><idno type="pmc">2311316</idno><idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2311316</idno><idno type="RBID">PMC:2311316</idno><idno type="doi">10.1186/1471-2407-8-66</idno><date when="2008">2008</date><idno type="wicri:Area/Pmc/Corpus">001B05</idno><idno type="wicri:explorRef" wicri:stream="Pmc" wicri:step="Corpus" wicri:corpus="PMC">001B05</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">New models and online calculator for predicting non-sentinel lymph node status in sentinel lymph node positive breast cancer patients</title><author><name sortKey="Kohrt, Holbrook E" sort="Kohrt, Holbrook E" uniqKey="Kohrt H" first="Holbrook E" last="Kohrt">Holbrook E. Kohrt</name><affiliation><nlm:aff id="I1">Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Olshen, Richard A" sort="Olshen, Richard A" uniqKey="Olshen R" first="Richard A" last="Olshen">Richard A. Olshen</name><affiliation><nlm:aff id="I2">Department of Health Research and Policy-Biostatistics, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation><affiliation><nlm:aff id="I3">Departments of Statistics and Electrical Engineering, Stanford University, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Bermas, Honnie R" sort="Bermas, Honnie R" uniqKey="Bermas H" first="Honnie R" last="Bermas">Honnie R. Bermas</name><affiliation><nlm:aff id="I4">Department of Surgery, Northwestern University Feinberg School of Medicine and Lynn Sage Comprehensive Breast Center, Northwestern Memorial Hospital, Chicago, IL, USA</nlm:aff></affiliation></author><author><name sortKey="Goodson, William H" sort="Goodson, William H" uniqKey="Goodson W" first="William H" last="Goodson">William H. Goodson</name><affiliation><nlm:aff id="I5">Department of Surgery, California Pacific Medical Center, San Francisco, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Wood, Douglas J" sort="Wood, Douglas J" uniqKey="Wood D" first="Douglas J" last="Wood">Douglas J. Wood</name><affiliation><nlm:aff id="I2">Department of Health Research and Policy-Biostatistics, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Henry, Solomon" sort="Henry, Solomon" uniqKey="Henry S" first="Solomon" last="Henry">Solomon Henry</name><affiliation><nlm:aff id="I2">Department of Health Research and Policy-Biostatistics, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Rouse, Robert V" sort="Rouse, Robert V" uniqKey="Rouse R" first="Robert V" last="Rouse">Robert V. Rouse</name><affiliation><nlm:aff id="I6">Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Bailey, Lisa" sort="Bailey, Lisa" uniqKey="Bailey L" first="Lisa" last="Bailey">Lisa Bailey</name><affiliation><nlm:aff id="I7">Department of Surgery, Alta Bates Summit Medical Center, Berkeley, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Philben, Vicki J" sort="Philben, Vicki J" uniqKey="Philben V" first="Vicki J" last="Philben">Vicki J. Philben</name><affiliation><nlm:aff id="I8">Department of Surgery, Mercy Medical Center, Redding, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Dirbas, Frederick M" sort="Dirbas, Frederick M" uniqKey="Dirbas F" first="Frederick M" last="Dirbas">Frederick M. Dirbas</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Dunn, Jocelyn J" sort="Dunn, Jocelyn J" uniqKey="Dunn J" first="Jocelyn J" last="Dunn">Jocelyn J. Dunn</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Johnson, Denise L" sort="Johnson, Denise L" uniqKey="Johnson D" first="Denise L" last="Johnson">Denise L. Johnson</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Wapnir, Irene L" sort="Wapnir, Irene L" uniqKey="Wapnir I" first="Irene L" last="Wapnir">Irene L. Wapnir</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Carlson, Robert W" sort="Carlson, Robert W" uniqKey="Carlson R" first="Robert W" last="Carlson">Robert W. Carlson</name><affiliation><nlm:aff id="I1">Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Stockdale, Frank E" sort="Stockdale, Frank E" uniqKey="Stockdale F" first="Frank E" last="Stockdale">Frank E. Stockdale</name><affiliation><nlm:aff id="I1">Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author><author><name sortKey="Hansen, Nora M" sort="Hansen, Nora M" uniqKey="Hansen N" first="Nora M" last="Hansen">Nora M. Hansen</name><affiliation><nlm:aff id="I4">Department of Surgery, Northwestern University Feinberg School of Medicine and Lynn Sage Comprehensive Breast Center, Northwestern Memorial Hospital, Chicago, IL, USA</nlm:aff></affiliation></author><author><name sortKey="Jeffrey, Stefanie S" sort="Jeffrey, Stefanie S" uniqKey="Jeffrey S" first="Stefanie S" last="Jeffrey">Stefanie S. Jeffrey</name><affiliation><nlm:aff id="I9">Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</nlm:aff></affiliation></author></analytic><series><title level="j">BMC Cancer</title><idno type="eISSN">1471-2407</idno><imprint><date when="2008">2008</date></imprint></series></biblStruct></sourceDesc></fileDesc><profileDesc><textClass></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en"><sec><title>Background</title><p>Current practice is to perform a completion axillary lymph node dissection (ALND) for breast cancer patients with tumor-involved sentinel lymph nodes (SLNs), although fewer than half will have non-sentinel node (NSLN) metastasis. Our goal was to develop new models to quantify the risk of NSLN metastasis in SLN-positive patients and to compare predictive capabilities to another widely used model.</p></sec><sec sec-type="methods"><title>Methods</title><p>We constructed three models to predict NSLN status: recursive partitioning with receiver operating characteristic curves (RP-ROC), boosted Classification and Regression Trees (CART), and multivariate logistic regression (MLR) informed by CART. Data were compiled from a multicenter Northern California and Oregon database of 784 patients who prospectively underwent SLN biopsy and completion ALND. We compared the predictive abilities of our best model and the Memorial Sloan-Kettering Breast Cancer Nomogram (Nomogram) in our dataset and an independent dataset from Northwestern University.</p></sec><sec><title>Results</title><p>285 patients had positive SLNs, of which 213 had known angiolymphatic invasion status and 171 had complete pathologic data including hormone receptor status. 264 (93%) patients had limited SLN disease (micrometastasis, 70%, or isolated tumor cells, 23%). 101 (35%) of all SLN-positive patients had tumor-involved NSLNs. Three variables (tumor size, angiolymphatic invasion, and SLN metastasis size) predicted risk in all our models. RP-ROC and boosted CART stratified patients into four risk levels. MLR informed by CART was most accurate. Using two composite predictors calculated from three variables, MLR informed by CART was more accurate than the Nomogram computed using eight predictors. In our dataset, area under ROC curve (AUC) was 0.83/0.85 for MLR (n = 213/n = 171) and 0.77 for Nomogram (n = 171). When applied to an independent dataset (n = 77), AUC was 0.74 for our model and 0.62 for Nomogram. The composite predictors in our model were the product of angiolymphatic invasion and size of SLN metastasis, and the product of tumor size and square of SLN metastasis size.</p></sec><sec><title>Conclusion</title><p>We present a new model developed from a community-based SLN database that uses only three rather than eight variables to achieve higher accuracy than the Nomogram for predicting NSLN status in two different datasets. </p></sec></div></front><back><div1 type="bibliography"><listBibl><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct></listBibl></div1></back></TEI><pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">BMC Cancer</journal-id><journal-title>BMC Cancer</journal-title><issn pub-type="epub">1471-2407</issn><publisher><publisher-name>BioMed Central</publisher-name></publisher></journal-meta><article-meta><article-id pub-id-type="pmid">18315887</article-id><article-id pub-id-type="pmc">2311316</article-id><article-id pub-id-type="publisher-id">1471-2407-8-66</article-id><article-id pub-id-type="doi">10.1186/1471-2407-8-66</article-id><article-categories><subj-group subj-group-type="heading"><subject>Research Article</subject></subj-group></article-categories><title-group><article-title>New models and online calculator for predicting non-sentinel lymph node status in sentinel lymph node positive breast cancer patients</article-title></title-group><contrib-group><contrib id="A1" contrib-type="author"><name><surname>Kohrt</surname><given-names>Holbrook E</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>kohrt@stanford.edu</email></contrib><contrib id="A2" contrib-type="author"><name><surname>Olshen</surname><given-names>Richard A</given-names></name><xref ref-type="aff" rid="I2">2</xref><xref ref-type="aff" rid="I3">3</xref><email>olshen@stanford.edu</email></contrib><contrib id="A3" contrib-type="author"><name><surname>Bermas</surname><given-names>Honnie R</given-names></name><xref ref-type="aff" rid="I4">4</xref><email>hbermas@nmh.org</email></contrib><contrib id="A4" contrib-type="author"><name><surname>Goodson</surname><given-names>William H</given-names></name><xref ref-type="aff" rid="I5">5</xref><email>goodsow@sutterhealth.org</email></contrib><contrib id="A5" contrib-type="author"><name><surname>Wood</surname><given-names>Douglas J</given-names></name><xref ref-type="aff" rid="I2">2</xref><email>djwood@stanford.edu</email></contrib><contrib id="A6" contrib-type="author"><name><surname>Henry</surname><given-names>Solomon</given-names></name><xref ref-type="aff" rid="I2">2</xref><email>shenry@stanford.edu</email></contrib><contrib id="A7" contrib-type="author"><name><surname>Rouse</surname><given-names>Robert V</given-names></name><xref ref-type="aff" rid="I6">6</xref><email>rouse@stanford.edu</email></contrib><contrib id="A8" contrib-type="author"><name><surname>Bailey</surname><given-names>Lisa</given-names></name><xref ref-type="aff" rid="I7">7</xref><email>baileylisa@msn.com</email></contrib><contrib id="A9" contrib-type="author"><name><surname>Philben</surname><given-names>Vicki J</given-names></name><xref ref-type="aff" rid="I8">8</xref><email>vphilben@reddingbreastcare.com</email></contrib><contrib id="A10" contrib-type="author"><name><surname>Dirbas</surname><given-names>Frederick M</given-names></name><xref ref-type="aff" rid="I9">9</xref><email>dirbas@stanford.edu</email></contrib><contrib id="A11" contrib-type="author"><name><surname>Dunn</surname><given-names>Jocelyn J</given-names></name><xref ref-type="aff" rid="I9">9</xref><email>jocelyndunn@sbcglobal.net</email></contrib><contrib id="A12" contrib-type="author"><name><surname>Johnson</surname><given-names>Denise L</given-names></name><xref ref-type="aff" rid="I9">9</xref><email>mudlj@stanford.edu</email></contrib><contrib id="A13" contrib-type="author"><name><surname>Wapnir</surname><given-names>Irene L</given-names></name><xref ref-type="aff" rid="I9">9</xref><email>wapnir@stanford.edu</email></contrib><contrib id="A14" contrib-type="author"><name><surname>Carlson</surname><given-names>Robert W</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>rcarlson@stanford.edu</email></contrib><contrib id="A15" contrib-type="author"><name><surname>Stockdale</surname><given-names>Frank E</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>stockdale@stanford.edu</email></contrib><contrib id="A16" contrib-type="author"><name><surname>Hansen</surname><given-names>Nora M</given-names></name><xref ref-type="aff" rid="I4">4</xref><email>nhansen@nmh.org</email></contrib><contrib id="A17" corresp="yes" contrib-type="author"><name><surname>Jeffrey</surname><given-names>Stefanie S</given-names></name><xref ref-type="aff" rid="I9">9</xref><email>ssj@stanford.edu</email></contrib><contrib id="A18" contrib-type="author"><collab>The Bay Area SLN Study</collab></contrib></contrib-group><aff id="I1"><label>1</label>Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA</aff><aff id="I2"><label>2</label>Department of Health Research and Policy-Biostatistics, Stanford University School of Medicine, Stanford, CA, USA</aff><aff id="I3"><label>3</label>Departments of Statistics and Electrical Engineering, Stanford University, Stanford, CA, USA</aff><aff id="I4"><label>4</label>Department of Surgery, Northwestern University Feinberg School of Medicine and Lynn Sage Comprehensive Breast Center, Northwestern Memorial Hospital, Chicago, IL, USA</aff><aff id="I5"><label>5</label>Department of Surgery, California Pacific Medical Center, San Francisco, CA, USA</aff><aff id="I6"><label>6</label>Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA</aff><aff id="I7"><label>7</label>Department of Surgery, Alta Bates Summit Medical Center, Berkeley, CA, USA</aff><aff id="I8"><label>8</label>Department of Surgery, Mercy Medical Center, Redding, CA, USA</aff><aff id="I9"><label>9</label>Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA</aff><pub-date pub-type="collection"><year>2008</year></pub-date><pub-date pub-type="epub"><day>4</day><month>3</month><year>2008</year></pub-date><volume>8</volume><fpage>66</fpage><lpage>66</lpage><ext-link ext-link-type="uri" xlink:href="http://www.biomedcentral.com/1471-2407/8/66"></ext-link><history><date date-type="received"><day>28</day><month>5</month><year>2007</year></date><date date-type="accepted"><day>4</day><month>3</month><year>2008</year></date></history><permissions><copyright-statement>Copyright © 2008 Kohrt et al; licensee BioMed Central Ltd.</copyright-statement><copyright-year>2008</copyright-year><copyright-holder>Kohrt et al; licensee BioMed Central Ltd.</copyright-holder><license license-type="open-access" xlink:href="http://creativecommons.org/licenses/by/2.0"><p>This is an Open Access article distributed under the terms of the Creative Commons Attribution License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/2.0"></ext-link>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</p><pmc-comment>
Kohrt
E
Holbrook
kohrt@stanford.edu
New models and online calculator for predicting non-sentinel lymph node status in sentinel lymph node positive breast cancer patients
2008 BMC Cancer 8(1): 66-. (2008) 1471-2407(2008)8:1<66> urn:ISSN:1471-2407
</license></permissions><abstract><sec><title>Background</title><p>Current practice is to perform a completion axillary lymph node dissection (ALND) for breast cancer patients with tumor-involved sentinel lymph nodes (SLNs), although fewer than half will have non-sentinel node (NSLN) metastasis. Our goal was to develop new models to quantify the risk of NSLN metastasis in SLN-positive patients and to compare predictive capabilities to another widely used model.</p></sec><sec sec-type="methods"><title>Methods</title><p>We constructed three models to predict NSLN status: recursive partitioning with receiver operating characteristic curves (RP-ROC), boosted Classification and Regression Trees (CART), and multivariate logistic regression (MLR) informed by CART. Data were compiled from a multicenter Northern California and Oregon database of 784 patients who prospectively underwent SLN biopsy and completion ALND. We compared the predictive abilities of our best model and the Memorial Sloan-Kettering Breast Cancer Nomogram (Nomogram) in our dataset and an independent dataset from Northwestern University.</p></sec><sec><title>Results</title><p>285 patients had positive SLNs, of which 213 had known angiolymphatic invasion status and 171 had complete pathologic data including hormone receptor status. 264 (93%) patients had limited SLN disease (micrometastasis, 70%, or isolated tumor cells, 23%). 101 (35%) of all SLN-positive patients had tumor-involved NSLNs. Three variables (tumor size, angiolymphatic invasion, and SLN metastasis size) predicted risk in all our models. RP-ROC and boosted CART stratified patients into four risk levels. MLR informed by CART was most accurate. Using two composite predictors calculated from three variables, MLR informed by CART was more accurate than the Nomogram computed using eight predictors. In our dataset, area under ROC curve (AUC) was 0.83/0.85 for MLR (n = 213/n = 171) and 0.77 for Nomogram (n = 171). When applied to an independent dataset (n = 77), AUC was 0.74 for our model and 0.62 for Nomogram. The composite predictors in our model were the product of angiolymphatic invasion and size of SLN metastasis, and the product of tumor size and square of SLN metastasis size.</p></sec><sec><title>Conclusion</title><p>We present a new model developed from a community-based SLN database that uses only three rather than eight variables to achieve higher accuracy than the Nomogram for predicting NSLN status in two different datasets. </p></sec></abstract></article-meta></front><body><sec><title>Background</title><p>Current practice guidelines recommend a completion axillary lymph node dissection for breast cancer patients whose SLN contains metastatic tumor [<xref ref-type="bibr" rid="B1">1</xref>-<xref ref-type="bibr" rid="B3">3</xref>]. The risk of morbidity that accompanies completion ALND seems justified for patients with NSLN metastases, because they would undergo excision of residual cancer [<xref ref-type="bibr" rid="B4">4</xref>]. However, 50 to 65% of patients with tumor-involved SLNs do not have additional nodal metastasis [<xref ref-type="bibr" rid="B5">5</xref>,<xref ref-type="bibr" rid="B6">6</xref>]. For them, ALND offers no clear therapeutic benefit, provides no further information for staging, and increases the cost of medical care. Further, completion ALND is associated with substantial morbidity affecting up to 39% of patients, with a nearly three-fold increased risk of lymphedema or regional sensory loss [<xref ref-type="bibr" rid="B7">7</xref>-<xref ref-type="bibr" rid="B9">9</xref>]. Identifying SLN-positive patients without NSLN metastases who could forgo completion ALND would improve the quality of life and reduce costs for the majority of women with new diagnoses of breast cancer.</p><p>Previous investigations have not identified predictors of NSLN status with accuracy sufficient to change clinical practice. This failure may be due to limited sample sizes or single institution studies [<xref ref-type="bibr" rid="B5">5</xref>,<xref ref-type="bibr" rid="B6">6</xref>,<xref ref-type="bibr" rid="B10">10</xref>]. The majority of prior investigations include sample sizes of less than two hundred subjects, with the challenges of dealing with small sample sizes leading to decreased predictive accuracy when applied to the general population [<xref ref-type="bibr" rid="B5">5</xref>,<xref ref-type="bibr" rid="B6">6</xref>,<xref ref-type="bibr" rid="B10">10</xref>-<xref ref-type="bibr" rid="B12">12</xref>]. However in 2003 Van Zee <italic>et al. </italic>proposed a nomogram to predict risk of NSLN metastasis based on an accrued population of 1075 cases of primary invasive breast cancer [<xref ref-type="bibr" rid="B13">13</xref>]. The Memorial Sloan-Kettering Cancer Center (MSKCC) Breast Cancer Nomogram (Nomogram) has since been successfully applied internationally and become the most commonly used predictive model for NSLN involvement [<xref ref-type="bibr" rid="B14">14</xref>]. Use of a predictive nomogram has been shown to be superior to expert opinion, to improve clinical decision making, and to be partially responsible for the decreasing frequency of ALNDs performed [<xref ref-type="bibr" rid="B15">15</xref>,<xref ref-type="bibr" rid="B16">16</xref>]. However, use of the Nomogram is limited by its complexity, and inability to be applied if not all patient characteristics are known [<xref ref-type="bibr" rid="B17">17</xref>]. Although the Nomogram was based on a large sample size, its reported predictive accuracy and its generalizability to patient populations with dissimilar tumor characteristics or to non-academic, non-quaternary care hospitals has been questioned [<xref ref-type="bibr" rid="B17">17</xref>-<xref ref-type="bibr" rid="B19">19</xref>].</p><p>Our goal was to identify characteristics of patients and their tumors that predict NSLN status within the Bay Area SLN Database, comprised of diverse patient populations from one academic and 15 community-based medical centers in Northern California and Oregon. We constructed three new models and contrasted their performance with the Nomogram. We provide a model that has simpler input than the Nomogram and shows higher accuracy for our diverse patient population and for another population of SLN-positive patients with different patient characteristics from Northwestern University. We have created an internet-based calculator, the Stanford Online Calculator, for validation testing and clinical application.</p></sec><sec sec-type="methods"><title>Methods</title><sec><title>Study patients</title><p>The <italic>Bay Area SLN Study for Detection of Axillary Metastasis in Breast Cancer </italic>is a multi-institutional collaboration involving 16 institutions in the Greater Bay Area of Northern California and Oregon, of which 15 are community hospitals. A total of 1,040 patients underwent SLN biopsy for biopsy-proven breast cancer between 1996 and 2002. After excluding 256 patients (criteria shown in Additional file <xref ref-type="supplementary-material" rid="S1">1</xref>), we analyzed 784 prospectively accrued subjects with primary invasive breast carcinoma and clinically negative axilla who underwent SLN biopsy with completion axillary lymph node dissection. 285 (36.4%) had tumor-involved SLNs. Among the 285 SLN-positive patients, 213 had pathologic information regarding presence or absence of angiolymphatic invasion (lymphovascular invasion, LVI); 171 patients had complete pathologic information on both angiolymphatic invasion and hormone receptor status. The <italic>Northwestern </italic>test dataset was compiled by chart review of all patients who underwent a SLN biopsy at Northwestern Memorial Hospital in Chicago, IL, between 2002 and 2006. It is comprised of 77 consecutively identified sentinel node positive patients with invasive breast cancer who underwent completion ALND and had complete pathologic information on tumor type, tumor size, tumor grade, hormone receptor status, HER2/<italic>neu </italic>status, angiolymphatic invasion status, number of nodes removed, and size of sentinel node metastases. Inclusion and exclusion criteria are similar to that outlined for the Stanford patients in Additional file <xref ref-type="supplementary-material" rid="S1">1</xref>. The Northwestern database was compiled by physicians not involved in generation of the predictive models. The Bay Area SLN study was performed under a protocol approved by the Stanford University Administrative Panel on Human Subjects in Medical Research and the Institutional Review Boards of each participating institution. An independent protocol was approved by the Institutional Review Board of Northwestern University for retrospective chart review and data collection to test the Stanford Online Calculator and MSKCC Nomogram.</p></sec><sec><title>SLN biopsy and pathological evaluation</title><p>SLN biopsy has been described previously [<xref ref-type="bibr" rid="B20">20</xref>]. The SLN was identified using peritumoral injection of 1% isosulfan blue dye, filtered <sup>99m</sup>Tc sulfur colloid radioactive tracer, or both, as decided by the operating surgeon. All lymph nodes that were blue and/or focally radioactive and/or suspicious by intraoperative palpation were denoted SLNs. All SLNs were evaluated by step-sectioning with hematoxylin and eosin (H&E) staining; in the Bay Area SLN study, SLNs without metastasis detectable by H&E underwent staining by immunohistochemistry (IHC) [<xref ref-type="bibr" rid="B21">21</xref>]. IHC was performed on at least four levels of the SLN using anti-keratin antibodies AE1 and CAM5.2. One pathologist directed and interpreted IHC studies on every SLN excised at 14 of the 16 participating institutions in the Bay Area SLN Study. NSLNs were evaluated by H&E only, without serial sectioning. In the Northwestern series, negative SLNs did not undergo IHC testing and individual tumor cells or clusters were identified on H&E only.</p></sec><sec><title>Statistical analyses</title><p>Thirteen characteristics were studied individually for predicting NSLN status: patient age, tumor histology, tumor size (as a continuous variable and as T size by 6<sup>th </sup>edition AJCC criteria), tumor grade [<xref ref-type="bibr" rid="B22">22</xref>], estrogen receptor (ER) status, progesterone receptor (PR) status, HER2/<italic>neu </italic>status, presence of angiolymphatic invasion, number of SLNs excised, number of positive SLNs, size of nodal metastasis (recorded according to revised 6<sup>th </sup>edition AJCC criteria) [<xref ref-type="bibr" rid="B23">23</xref>], and method of detecting nodal metastasis (H&E or IHC). Univariate testing was done with χ<sup>2 </sup>statistics and Wilcoxon rank sums. For multivariate analyses, tree-based classification and logistic regression were performed [<xref ref-type="bibr" rid="B24">24</xref>,<xref ref-type="bibr" rid="B25">25</xref>]. Recognizing that some characteristics can be interdependent, we performed multivariate analyses with two approaches whereby interactions among variables are emphasized: recursive partitioning via receiver operating characteristic (RP-ROC) [<xref ref-type="bibr" rid="B26">26</xref>] curves and (boosted) classification and regression trees (CART<sup>®</sup>) [<xref ref-type="bibr" rid="B24">24</xref>,<xref ref-type="bibr" rid="B27">27</xref>,<xref ref-type="bibr" rid="B28">28</xref>].</p><p>RP-ROC uses the relationship of sensitivity and specificity to calculate the "best value" of each variable for predicting NSLN status. It then chooses the variable with best value. Successive partitioning permits use of ROC curves to compare predictive accuracy and best cut point on "best selected variable." Partitioning of the population into subgroups continues until only patients with or without NSLN metastases are segregated to the group, or until the putative <italic>p </italic>value of the split exceeds 0.01. RP-ROC was performed as is described in detail by Kraemer [<xref ref-type="bibr" rid="B26">26</xref>] (software available from Sierra-Pacific MIRECC [<xref ref-type="bibr" rid="B29">29</xref>]).</p><p>CART as we applied it uses both cross-validation and voting methods (boosting) to assess the stability and improve the accuracy of the final model [<xref ref-type="bibr" rid="B24">24</xref>,<xref ref-type="bibr" rid="B27">27</xref>], (software available from Salford Systems, v5 [<xref ref-type="bibr" rid="B30">30</xref>]). Splits are chosen by what is termed the Gini criterion, whose goal is to render nodes of the tree as "pure" as possible in terms of positive or negative NSLN status. Boosting is a method designed to focus on "hard to classify" observations. In all classifications, there is dependence on the products by class of priors and costs of misclassification. For all classification trees, mixed priors (an average of equal priors and prevalence-based priors) were used. After surveying eighteen breast surgeons expert in SLN biopsy and not associated with this study, the costs of a false-positive and false-negative NSLN were set at 3 and 10, respectively.</p><p>A third technique, multivariate logistic regression (MLR) informed by CART, was performed with variable selection based on paths from the root to the five terminal nodes of unboosted CART [<xref ref-type="bibr" rid="B31">31</xref>]. Odds ratios were calculated individually for all terms that were candidates for inclusion in subsequent analyses. Those terms retained were entered into the MLR by forward selection based on the likelihood ratio. Wald statistics and odds ratios were determined for variables significant at putative <italic>p </italic>< 0.01 within the regression model [<xref ref-type="bibr" rid="B32">32</xref>]. A cutoff p < 0.01 was chosen in the interest of our ending with a focused, concise, predictive model.</p><p>In constructing the predictive models of NSLN status, we used tumor characteristics that were significant by univariate testing (Table <xref ref-type="table" rid="T1">1</xref>): tumor size, tumor grade, ER status, PR status, angiolymphatic invasion, size of SLN metastasis, and SLN metastasis identification method. Statistical modeling of NSLN status allowed calculation of both the predictive capacity of significant variables and the critical interactions between and among variables, such as increasing angiolymphatic invasion with increasing tumor size. All models used identical variables, although not identical patients. RP-ROC requires complete data, where no values of features are missing, whereas CART does not. Instead, CART relies on the subtle notion of "surrogate split" [<xref ref-type="bibr" rid="B24">24</xref>]. Thus, boosted CART analyses were performed on all 285 SLN-positive patients as well as subsets with more complete information, while RP-ROC and MLR analyses were performed on the 213 patients with complete data for angiolymphatic invasion and on the 171 patients with complete data for angiolymphatic invasion and hormone receptor status.</p><table-wrap position="float" id="T1"><label>Table 1</label><caption><p>Characteristics of NSLN- and NSLN+ cases among SLN+ patients (Bay Area SLN Database).</p></caption><table frame="hsides" rules="groups"><thead><tr><td></td><td align="center" colspan="4"><bold>Tumor-free NSLN</bold></td><td align="center" colspan="4"><bold>Tumor-involved NSLN</bold></td><td align="center"><italic>Univariate P </italic>∫</td><td align="center"><italic>Multivariate P </italic>∫∫</td><td align="center"><bold>%NSLN+ </bold>(NSLN+/SLN+)</td><td align="center" colspan="2"><bold>TOTAL SLN+</bold></td></tr></thead><tbody><tr><td align="left"><bold><italic>Patient and Tumor Characteristics</italic></bold></td><td align="center">Number of Pts (n = 184)</td><td align="center"><italic>%</italic></td><td align="center">Mean</td><td align="center">SEM*</td><td align="center">Number of Pts (n = 101)</td><td align="center"><italic>%</italic></td><td align="center">Mean</td><td align="center">SEM*</td><td></td><td></td><td></td><td align="center">Number of Pts (n = 285)</td><td align="center"><italic>%</italic></td></tr><tr><td colspan="14"><hr></hr></td></tr><tr><td align="left"><bold>Patient Age (years)</bold></td><td></td><td></td><td align="center">55.8</td><td align="center">0.89</td><td></td><td></td><td align="center">53</td><td align="center">1.14</td><td align="center">0.084</td><td align="center">NA</td><td></td><td></td><td></td></tr><tr><td align="left"><bold>Tumor Type</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td align="left"> Infiltrating Ductal Carcinoma</td><td align="center">159</td><td align="center"><italic>86</italic></td><td></td><td></td><td align="center">87</td><td align="center"><italic>86</italic></td><td></td><td></td><td align="center">0.781</td><td align="center">NA</td><td align="center">35%</td><td align="center">246</td><td align="center"><italic>86.3</italic></td></tr><tr><td align="left"> Invasive Lobular Carcinoma</td><td align="center">18</td><td align="center"><italic>10</italic></td><td></td><td></td><td align="center">9</td><td align="center"><italic>9</italic></td><td></td><td></td><td></td><td></td><td align="center">33%</td><td align="center">27</td><td align="center"><italic>9.5</italic></td></tr><tr><td align="left"> Mixed Carcinoma</td><td align="center">6</td><td align="center"><italic>3</italic></td><td></td><td></td><td align="center">5</td><td align="center"><italic>5</italic></td><td></td><td></td><td></td><td></td><td align="center">45%</td><td align="center">11</td><td align="center"><italic>3.9</italic></td></tr><tr><td align="left"> Tubular Carcinoma</td><td align="center">1</td><td align="center"><italic>1</italic></td><td></td><td></td><td align="center">0</td><td align="center"><italic>0</italic></td><td></td><td></td><td></td><td></td><td align="center">0%</td><td align="center">1</td><td align="center"><italic>0.4</italic></td></tr><tr><td align="left"><bold>Tumor size (cm)</bold></td><td></td><td></td><td align="center">2.11</td><td align="center">0.1</td><td></td><td></td><td align="center">2.97</td><td align="center">0.18</td><td align="center"><0.001</td><td align="center"><0.001</td><td></td><td></td><td></td></tr><tr><td align="left"><bold>Tumor size (AJCC)</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td align="center">0.001</td><td align="center">0.045</td><td></td><td></td><td></td></tr><tr><td align="left"> T1</td><td align="center">117</td><td align="center"><italic>64</italic></td><td></td><td></td><td align="center">39</td><td align="center"><italic>39</italic></td><td></td><td></td><td></td><td></td><td align="center">25%</td><td align="center">156</td><td align="center"><italic>54.7</italic></td></tr><tr><td align="left"> T1a (mic)</td><td align="center">1</td><td align="center"><italic>1</italic></td><td></td><td></td><td align="center">0</td><td align="center"><italic>0</italic></td><td></td><td></td><td></td><td></td><td align="center">0%</td><td align="center">1</td><td align="center"><italic>0.4</italic></td></tr><tr><td align="left"> T1a</td><td align="center">6</td><td align="center"><italic>3</italic></td><td></td><td></td><td align="center">2</td><td align="center"><italic>2</italic></td><td></td><td></td><td></td><td></td><td align="center">25%</td><td align="center">8</td><td align="center"><italic>2.8</italic></td></tr><tr><td align="left"> T1b</td><td align="center">19</td><td align="center"><italic>10</italic></td><td></td><td></td><td align="center">3</td><td align="center"><italic>3</italic></td><td></td><td></td><td></td><td></td><td align="center">14%</td><td align="center">22</td><td align="center"><italic>7.7</italic></td></tr><tr><td align="left"> T1c</td><td align="center">91</td><td align="center"><italic>49</italic></td><td></td><td></td><td align="center">34</td><td align="center"><italic>34</italic></td><td></td><td></td><td></td><td></td><td align="center">27%</td><td align="center">125</td><td align="center"><italic>43.9</italic></td></tr><tr><td align="left"> T2</td><td align="center">59</td><td align="center"><italic>32</italic></td><td></td><td></td><td align="center">50</td><td align="center"><italic>50</italic></td><td></td><td></td><td></td><td></td><td align="center">46%</td><td align="center">109</td><td align="center"><italic>38.2</italic></td></tr><tr><td align="left"> T3</td><td align="center">8</td><td align="center"><italic>4</italic></td><td></td><td></td><td align="center">12</td><td align="center"><italic>12</italic></td><td></td><td></td><td></td><td></td><td align="center">60%</td><td align="center">20</td><td align="center"><italic>7.0</italic></td></tr><tr><td align="left"><bold>Tumor grade†</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td align="center">0.001</td><td align="center">0.736</td><td></td><td></td><td></td></tr><tr><td align="left"> G1: Nottingham combined histologic score 3–5</td><td align="center">68</td><td align="center"><italic>37</italic></td><td></td><td></td><td align="center">23</td><td align="center"><italic>23</italic></td><td></td><td></td><td></td><td></td><td align="center">25%</td><td align="center">91</td><td align="center"><italic>31.9</italic></td></tr><tr><td align="left"> G2: Nottingham combined histologic score 6–7</td><td align="center">81</td><td align="center"><italic>44</italic></td><td></td><td></td><td align="center">39</td><td align="center"><italic>39</italic></td><td></td><td></td><td></td><td></td><td align="center">33%</td><td align="center">120</td><td align="center"><italic>42.1</italic></td></tr><tr><td align="left"> G3: Nottingham combined histologic score 8–9</td><td align="center">35</td><td align="center"><italic>19</italic></td><td></td><td></td><td align="center">39</td><td align="center"><italic>39</italic></td><td></td><td></td><td></td><td></td><td align="center">53%</td><td align="center">74</td><td align="center"><italic>26.0</italic></td></tr><tr><td align="left"><bold>ER status</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td align="center">0.004</td><td align="center">0.079</td><td></td><td></td><td></td></tr><tr><td align="left"> Negative</td><td align="center">17</td><td align="center"><italic>9</italic></td><td></td><td></td><td align="center">21</td><td align="center"><italic>21</italic></td><td></td><td></td><td></td><td></td><td align="center">55%</td><td align="center">38</td><td align="center"><italic>13.3</italic></td></tr><tr><td align="left"> Positive</td><td align="center">134</td><td align="center"><italic>73</italic></td><td></td><td></td><td align="center">60</td><td align="center"><italic>59</italic></td><td></td><td></td><td></td><td></td><td align="center">31%</td><td align="center">194</td><td align="center"><italic>68.1</italic></td></tr><tr><td align="left"> Unknown</td><td align="center">33</td><td align="center"><italic>18</italic></td><td></td><td></td><td align="center">20</td><td align="center"><italic>20</italic></td><td></td><td></td><td></td><td></td><td align="center">38%</td><td align="center">53</td><td align="center"><italic>18.6</italic></td></tr><tr><td align="left"><bold>PR status</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td align="center">0.015</td><td align="center">0.869</td><td></td><td></td><td></td></tr><tr><td align="left"> Negative</td><td align="center">35</td><td align="center"><italic>19</italic></td><td></td><td></td><td align="center">31</td><td align="center"><italic>31</italic></td><td></td><td></td><td></td><td></td><td align="center">47%</td><td align="center">66</td><td align="center"><italic>23.2</italic></td></tr><tr><td align="left"> Positive</td><td align="center">116</td><td align="center"><italic>63</italic></td><td></td><td></td><td align="center">50</td><td align="center"><italic>50</italic></td><td></td><td></td><td></td><td></td><td align="center">30%</td><td align="center">166</td><td align="center"><italic>58.2</italic></td></tr><tr><td align="left"> Unknown</td><td align="center">33</td><td align="center"><italic>18</italic></td><td></td><td></td><td align="center">20</td><td align="center"><italic>20</italic></td><td></td><td></td><td></td><td></td><td align="center">38%</td><td align="center">53</td><td align="center"><italic>18.6</italic></td></tr><tr><td align="left"><bold>HER2/neu expression</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td align="center">0.256</td><td align="center">NA</td><td></td><td></td><td></td></tr><tr><td align="left"> Not overexpressed, 0+ or 1+</td><td align="center">81</td><td align="center"><italic>44</italic></td><td></td><td></td><td align="center">38</td><td align="center"><italic>38</italic></td><td></td><td></td><td></td><td></td><td align="center">32%</td><td align="center">119</td><td align="center"><italic>41.8</italic></td></tr><tr><td align="left"> Equivocal, weak overexpression, 2+</td><td align="center">2</td><td align="center"><italic>1</italic></td><td></td><td></td><td align="center">2</td><td align="center"><italic>2</italic></td><td></td><td></td><td></td><td></td><td align="center">50%</td><td align="center">4</td><td align="center"><italic>1.4</italic></td></tr><tr><td align="left"> Overexpressed, 3+</td><td align="center">29</td><td align="center"><italic>16</italic></td><td></td><td></td><td align="center">23</td><td align="center"><italic>23</italic></td><td></td><td></td><td></td><td></td><td align="center">44%</td><td align="center">52</td><td align="center"><italic>18.2</italic></td></tr><tr><td align="left"> Unknown</td><td align="center">72</td><td align="center"><italic>39</italic></td><td></td><td></td><td align="center">38</td><td align="center"><italic>38</italic></td><td></td><td></td><td></td><td></td><td align="center">35%</td><td align="center">110</td><td align="center"><italic>38.6</italic></td></tr><tr><td align="left"><bold>Angiolymphatic invasion</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td align="center"><0.001</td><td align="center"><0.001</td><td></td><td></td><td></td></tr><tr><td align="left"> None</td><td align="center">95</td><td align="center"><italic>52</italic></td><td></td><td></td><td align="center">23</td><td align="center"><italic>23</italic></td><td></td><td></td><td></td><td></td><td align="center">19%</td><td align="center">118</td><td align="center"><italic>41.4</italic></td></tr><tr><td align="left"> Present</td><td align="center">25</td><td align="center"><italic>14</italic></td><td></td><td></td><td align="center">70</td><td align="center"><italic>69</italic></td><td></td><td></td><td></td><td></td><td align="center">74%</td><td align="center">95</td><td align="center"><italic>33.3</italic></td></tr><tr><td align="left"> Unknown</td><td align="center">64</td><td align="center"><italic>35</italic></td><td></td><td></td><td align="center">8</td><td align="center"><italic>8</italic></td><td></td><td></td><td></td><td></td><td align="center">11%</td><td align="center">72</td><td align="center"><italic>25.3</italic></td></tr><tr><td align="left"><bold><italic>Sentinel Lymph Node Characteristics</italic></bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td align="left"><bold>No. SLNs Removed</bold></td><td></td><td></td><td align="center">1.96</td><td align="center">0.07</td><td></td><td></td><td align="center">1.87</td><td align="center">0.09</td><td align="center">0.511</td><td align="center">NA</td><td></td><td></td><td></td></tr><tr><td align="left"> = 1</td><td align="center">71</td><td align="center"><italic>39</italic></td><td></td><td></td><td align="center">42</td><td align="center"><italic>42</italic></td><td></td><td></td><td align="center">0.806</td><td></td><td align="center">37%</td><td align="center">113</td><td align="center"><italic>39.6</italic></td></tr><tr><td align="left"> = 2</td><td align="center">67</td><td align="center"><italic>36</italic></td><td></td><td></td><td align="center">37</td><td align="center"><italic>37</italic></td><td></td><td></td><td></td><td></td><td align="center">36%</td><td align="center">104</td><td align="center"><italic>36.5</italic></td></tr><tr><td align="left"> >2</td><td align="center">46</td><td align="center"><italic>25</italic></td><td></td><td></td><td align="center">22</td><td align="center"><italic>22</italic></td><td></td><td></td><td></td><td></td><td align="center">32%</td><td align="center">68</td><td align="center"><italic>23.9</italic></td></tr><tr><td align="left"><bold>No. SLNs Tumor-involved</bold></td><td></td><td></td><td align="center">1.33</td><td align="center">0.05</td><td></td><td></td><td align="center">1.39</td><td align="center">0.07</td><td align="center">0.426</td><td align="center">NA</td><td></td><td></td><td></td></tr><tr><td align="left"> = 1</td><td align="center">137</td><td align="center"><italic>73</italic></td><td></td><td></td><td align="center">71</td><td align="center"><italic>70</italic></td><td></td><td></td><td align="center">0.679</td><td></td><td align="center">34%</td><td align="center">208</td><td align="center"><italic>73.0</italic></td></tr><tr><td align="left"> = 2</td><td align="center">38</td><td align="center"><italic>21</italic></td><td></td><td></td><td align="center">23</td><td align="center"><italic>23</italic></td><td></td><td></td><td></td><td></td><td align="center">38%</td><td align="center">61</td><td align="center"><italic>21.4</italic></td></tr><tr><td align="left"> >2</td><td align="center">9</td><td align="center"><italic>6</italic></td><td></td><td></td><td align="center">7</td><td align="center"><italic>7</italic></td><td></td><td></td><td></td><td></td><td align="center">44%</td><td align="center">16</td><td align="center"><italic>5.6</italic></td></tr><tr><td align="left"><bold>Size of SLN metastases§</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td align="center"><0.001</td><td align="center"><0.001</td><td></td><td></td><td></td></tr><tr><td align="left"> Isolated tumor cells or clusters ≤ 0.2 mm</td><td align="center">61</td><td align="center"><italic>33</italic></td><td></td><td></td><td align="center">3</td><td align="center"><italic>3</italic></td><td></td><td></td><td></td><td></td><td align="center">4.7%</td><td align="center">64</td><td align="center"><italic>22.5</italic></td></tr><tr><td align="left"> Micrometastases, >0.2 mm to 2 mm</td><td align="center">117</td><td align="center"><italic>64</italic></td><td></td><td></td><td align="center">83</td><td align="center"><italic>82</italic></td><td></td><td></td><td></td><td></td><td align="center">42%</td><td align="center">200</td><td align="center"><italic>70.2</italic></td></tr><tr><td align="left"> Macrometastases, >2 mm</td><td align="center">6</td><td align="center"><italic>3</italic></td><td></td><td></td><td align="center">15</td><td align="center"><italic>15</italic></td><td></td><td></td><td></td><td></td><td align="center">71%</td><td align="center">21</td><td align="center"><italic>7.4</italic></td></tr><tr><td align="left"><bold>Sentinel lymph node metastases identification</bold></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td align="center"><0.001</td><td align="center">NA</td><td></td><td></td><td></td></tr><tr><td align="left"> Hematoxylin and eosin staining</td><td align="center">122</td><td align="center"><italic>61</italic></td><td></td><td></td><td align="center">98</td><td align="center"><italic>97</italic></td><td></td><td></td><td></td><td></td><td align="center">45%</td><td align="center">220</td><td align="center"><italic>77.2</italic></td></tr><tr><td align="left"> Immunohistochemistry</td><td align="center">62</td><td align="center"><italic>34</italic></td><td></td><td></td><td align="center">3</td><td align="center"><italic>3</italic></td><td></td><td></td><td></td><td></td><td align="center">4.6%</td><td align="center">65</td><td align="center"><italic>22.8</italic></td></tr></tbody></table><table-wrap-foot><p>∫ Univariate analyses calculated by χ 2 test and Wilcoxon rank sum test.</p><p>∫∫ Multivariate analyses calculated by logistic regression of significant factors by univariate analysis.</p><p>*SEM, standard error of the mean.</p><p>†Determined according to modified Scarff-Bloom-Richardson grading system.</p><p>§Determined according to AJCC criteria, 6th ed.</p><p>NA, variable not included in multivariate analysis (not an independent factor or not significant in univariate analysis)</p></table-wrap-foot></table-wrap><p>The MSKCC Breast Cancer Nomogram for Prediction of ALN Status [<xref ref-type="bibr" rid="B13">13</xref>] (Nomogram) was applied to our patient population and, to provide fair comparison, calculated for only the 171 patients with complete information on the eight variables required for its application (pathologic size of primary tumor, tumor type with nuclear grade if ductal, LVI, multifocality of primary tumor, ER status, method of detecting SLN metastasis, number of positive SLNs, and number of negative SLNs; a ninth variable, whether a frozen section was performed, was not applicable to our patients). ROC curves were constructed for the Nomogram and the other methods to compare the area under the curve (AUC). Internal validation was performed by 10-fold cross-validation, as previously described [<xref ref-type="bibr" rid="B27">27</xref>]. Data were divided at random into 10 parts, as equal as possible in size. CART (in this instance, but more generally any other procedure) was then computed successively for 9/10 of the data with the remaining piece held out as "test sample." This was repeated 10 times and results on the 10 test samples were averaged. Cross-validation is an internal validation method that estimates performance on subsequent subjects by eliminating bias that owes to using the same, or even a portion of the same, data for both modeling and testing. However, even with internal validation, bias and variability can be introduced into subsequent analyses if the prevalence of features that predict outcome is different in future datasets than in the dataset from which the model was developed. The differences in distribution of variables (and in synergistic interactions between variables) for an original and a subsequent test dataset impacts a model's performance on future datasets and applies both to our models and to that of the Nomogram. For this reason, we tested our model and the Nomogram on the Northwestern dataset that differed from our original dataset in its distribution of patient, tumor, and sentinel node variables.</p><p>ROC curves were constructed for the Nomogram and the MLR informed by CART model for the Bay Area SLN study dataset and the independent Northwestern dataset.</p><p>Statistical analyses were performed with R [<xref ref-type="bibr" rid="B33">33</xref>].</p></sec></sec><sec><title>Results</title><p>Table <xref ref-type="table" rid="T1">1</xref> and Additional files <xref ref-type="supplementary-material" rid="S2">2</xref> and <xref ref-type="supplementary-material" rid="S3">3</xref> describe in detail the SLN-negative and SLN-positive patients of the <italic>Bay Area SLN </italic>dataset. As expected, the incidence of SLN metastasis increased with increasing tumor size: 29% of T1, 51% of T2, and 80% of T3 tumors had SLN metastasis. As tumor size increased over 1 cm, the incidence of angiolymphatic invasion doubled for both SLN-positive and SLN-negative patients but was higher for SLN-positive patients (Additional file <xref ref-type="supplementary-material" rid="S3">3</xref>). Among all 784 patients, the total number of women with any axillary lymph node metastasis was 316 (40%), including 31 (9.8%) with a false negative SLN (Additional file <xref ref-type="supplementary-material" rid="S2">2</xref>).</p><p>Among SLN-positive cases, the average number of SLNs removed was 1.91, with metastatic disease limited to a single SLN in 73%. Among tumor-involved SLNs, 23% contained isolated tumor cells or clusters (ITCs, ≤ 0.2 mm); 70% contained micrometastases (>0.2 mm to 2 mm); and 7% contained macrometastases (>2 mm). All SLNs containing ITCs required IHC for detection. Only one of 200 cases with SLNs involved by micrometastasis was not observed on H&E and required IHC staining for identification. All 21 cases with SLN macrometastasis were identified by H&E staining (Additional file <xref ref-type="supplementary-material" rid="S2">2</xref>).</p><p>Of 285 patients with tumor-involved SLNs, 101 (35.4%) were found to have NSLN metastases, with tumor metastases to two or more NSLNs in the majority of cases (median number of positive NSLNs 2; mean 3.5; range 1–19) (Additional file <xref ref-type="supplementary-material" rid="S2">2</xref>). By univariate analyses, 8 variables were highly predictive of NSLN status: tumor size (in cm), tumor size by AJCC T classification, tumor grade, ER status, PR status, angiolymphatic invasion, size of SLN metastasis, and whether the nodal metastasis was identified by H&E or IHC (Table <xref ref-type="table" rid="T1">1</xref>). Of patients whose SLN was identified by H&E, 45% had NSLN metastases, whereas only 4.6% of patients whose SLN was identified by IHC had NSLN metastases (<italic>p </italic>< 0.001). Size of SLN metastasis and staining method for metastasis identification are highly correlated (<italic>p </italic>= 0.02, by χ<sup>2 </sup>testing) and therefore are not independent predictors of NSLN status. Thus, staining method for identifying tumor-involvement was not included in the multivariate analysis shown in Table <xref ref-type="table" rid="T1">1</xref>. By multivariate analysis, tumor size, angiolymphatic invasion, and size of SLN metastasis remained significantly predictive of NSLN status (<italic>p </italic>< 0.001 by unconditional testing). Of the 285 patients with SLN metastases, NSLN metastases were found in 25% of patients with T1 tumors; in 46% with T2 tumors; and in 60% with T3 tumors (Figure <xref ref-type="fig" rid="F1">1A</xref>). When angiolymphatic invasion was present, there was a 3.9-fold increase in NSLN metastases (74% vs. 19%, Figure <xref ref-type="fig" rid="F1">1B</xref>). Among patients with isolated tumor cells or clusters within the SLN, 4.7% had NSLN metastasis; whereas 42% of patients with micrometastasis and 71% with macrometastasis had NSLN involvement (Figure <xref ref-type="fig" rid="F1">1C</xref> and Table <xref ref-type="table" rid="T1">1</xref>).</p><fig position="float" id="F1"><label>Figure 1</label><caption><p>Fraction of patients in Bay Area SLN Database with and without NSLN metastases in relation to (A) tumor stage, (B) angiolymphatic invasion, and (C) size of SLN metastasis.</p></caption><graphic xlink:href="1471-2407-8-66-1"></graphic></fig><p>The models generated by RP-ROC (Figure <xref ref-type="fig" rid="F2">2A</xref>) and CART (Figure <xref ref-type="fig" rid="F2">2B</xref>, Additional files <xref ref-type="supplementary-material" rid="S4">4</xref> and <xref ref-type="supplementary-material" rid="S5">5</xref>) ultimately included tumor size, angiolymphatic invasion, and size of SLN metastasis. At the final split, likelihood of NSLN metastases partitioned into groups by level of risk. The significant predictors as selected by multivariate tree-based modeling were tested individually, as well as all iterations of predictors, in a MLR model. Variables entered were tumor size, angiolymphatic invasion, and size of SLN metastasis (Table <xref ref-type="table" rid="T2">2</xref>). Size of SLN metastases interacts with the status of angiolymphatic invasion; that is, the impact of the size of SLN metastases upon the presence or absence of NSLN metastases depends on whether there was angiolymphatic invasion. The tree suggests that one might enter angiolymphatic invasion (scored as 1 if present, 0 if absent) not only multiplied by SLN metastasis size to the first power, but also as the product of angiolymphatic invasion and the square of SLN metastasis size (scored as an ordinal variable with values of 1, 2, and 3 corresponding to the size classification of isolated tumor cells, micrometastasis, or macrometastasis). The MLR model identified two highly predictive composite variables: the product of angiolymphatic invasion and size of SLN metastasis (<italic>p </italic>< 0.0001, odds ratio of 4.73 with approximate 95% confidence interval 3.11–7.20) as well as the product of tumor size and squared size of SLN metastasis (<italic>p </italic>< 0.0001, odds ratio of 1.18 with 95% confidence interval 1.10–1.26). We emphasize that <italic>p</italic>-values are only approximate because CART was used as preprocessor to manufacturing the predictive variables. However, these <italic>p</italic>-values are so small, and the clinical logic so compelling, that we do not doubt their practical, let alone statistical, significance.</p><fig position="float" id="F2"><label>Figure 2</label><caption><p><bold>Tree diagrams for RP-ROC and CART.</bold> As CART is able to impute missing data, it was calculated for all SLN positive patients, n = 285. RP-ROC requires complete data and was calculated for patients with known angiolymphatic invasion status, n = 213 (Bay Area SLN Database).</p></caption><graphic xlink:href="1471-2407-8-66-2"></graphic></fig><table-wrap position="float" id="T2"><label>Table 2</label><caption><p>Multivariate Logistic Regression (MLR) analysis informed by CART for predicting NSLN metastasis among SLN+ patients (n = 213) (Bay Area SLN Database).</p></caption><table frame="hsides" rules="groups"><thead><tr><td align="left"><bold>Variable or Composite Variable</bold></td><td align="center"><bold>Significance</bold></td><td align="center"><bold>Wald Statistic</bold></td><td align="center"><bold>OR (95% CI)</bold></td></tr></thead><tbody><tr><td align="left">Angiolymphatic Invasion</td><td align="center">0.024</td><td align="center">a</td><td align="center">a</td></tr><tr><td align="left">Tumor Size</td><td align="center">0.508</td><td align="center">a</td><td align="center">a</td></tr><tr><td align="left">Size of SLN Metastasis</td><td align="center">0.173</td><td align="center">a</td><td align="center">a</td></tr><tr><td align="left">Angiolymphatic Invasion × Tumor Size</td><td align="center">0.166</td><td align="center">a</td><td align="center">a</td></tr><tr><td align="left">Angiolymphatic Invasion × Size of SLN Metastasis</td><td align="center"><0.0001</td><td align="center">52.7</td><td align="center">4.73 (3.11–7.20)</td></tr><tr><td align="left">Angiolymphatic Invasion × (Size of SLN Metastasis)<sup>2</sup></td><td align="center">0.041</td><td align="center">a</td><td align="center">a</td></tr><tr><td align="left">Tumor Size × Size of SLN Metastasis</td><td align="center">0.888</td><td align="center">a</td><td align="center">a</td></tr><tr><td align="left">Tumor Size × (Size of SLN Metastasis)<sup>2</sup></td><td align="center"><0.0001</td><td align="center">22.6</td><td align="center">1.18 (1.10–1.26)</td></tr><tr><td align="left">Tumor Size × Size of SLN Metastasis × Angiolymphatic Invasion</td><td align="center">0.471</td><td align="center">a</td><td align="center">a</td></tr><tr><td align="left">Tumor Size × (Size of SLN Metastasis)<sup>2 </sup>× Angiolymphatic Invasion</td><td align="center">0.761</td><td align="center">a</td><td align="center">a</td></tr></tbody></table><table-wrap-foot><p>OR: odds ratio; 95% CI: 95% confidence interval. <sup>a </sup>No Wald statistic or odds ratio calculated for variables not significant at <italic>P </italic>< 0.01 with respect to multivariate model</p></table-wrap-foot></table-wrap><p>Table <xref ref-type="table" rid="T3">3</xref> compares the sensitivities, specificities, and predictive accuracies of our three models, RP-ROC, boosted CART, and MLR, all computed with 10-fold cross validation [<xref ref-type="bibr" rid="B26">26</xref>]. As different models require different information, we evaluated models for the entire group (n = 285, only possible for CART) and subsets that contained complete information on angiolymphatic invasion (n = 213), and alternatively, on angiolymphatic invasion and ER status (n = 171). Cross-validated sensitivities/specificities of the three technologies for the group with known angiolymphatic invasion status (n = 213) were 79%/76% for RP-ROC, 88%/71% for boosted CART, and 78%/86% for MLR. Cross-validated specificity of boosted CART when inferred for the entire dataset (n = 285) was lower than when calculated using known values for angiolymphatic invasion (n = 213), suggesting that angiolymphatic invasion is informative in our dataset. This is supported by the continued selection of angiolymphatic invasion in CART modeling when patients have known angiolymphatic invasion status (n = 213) and known angiolymphatic status and ER status (n = 171) (Additional files <xref ref-type="supplementary-material" rid="S4">4</xref> and <xref ref-type="supplementary-material" rid="S5">5</xref>, respectively). Overall diagnostic accuracy, based on areas under the ROC curve [<xref ref-type="bibr" rid="B34">34</xref>] (AUC), for predicting NSLN metastasis among patients in our database was greatest by MLR (83% and 85%) for the subsets of patients for whom the computation was possible (n = 213 and n = 171, respectively). Further, we applied the Nomogram to our SLN-positive patients who had complete data available for entry of its eight variables (n = 171, all patients with known angiolymphatic invasion status and ER status). Figure <xref ref-type="fig" rid="F3">3</xref> shows a graph of the ROC curve that devolves from our MLR using our two composite variables (n = 213) and the ROC curve that devolves from the Nomogram (n = 171). Because much preprocessing has gone into our computations, <italic>p</italic>-values we might report (regarding a null hypothesis that the "true" areas under the curves are equal) would be suspect. However, the diagnostic accuracy or area under the curve (AUC) for our MLR is 83% (95% confidence interval 0.81–0.86), and the AUC for the Nomogram is 77% (95% confidence interval 0.73–0.81). When we use the same patients as used in the Nomogram for the MLR calculation (n = 171), our model achieves cross-validated AUC of 85% (95% confidence interval 0.81–0.89). Given that only three variables were used to calculate our MLR, the difference is noteworthy.</p><fig position="float" id="F3"><label>Figure 3</label><caption><p><bold>ROC curves for MLR informed by CART calculation in blue, AUC = 0.83, and Nomogram in green, AUC = 0.77, when applied to the Bay Area SLN Database.</bold> Note that MLR informed by CART calculation was done for larger group of patients (n = 213). When it was performed for the same patient group as the Nomogram (n = 171), AUC increased to 0.85.</p></caption><graphic xlink:href="1471-2407-8-66-3"></graphic></fig><table-wrap position="float" id="T3"><label>Table 3</label><caption><p>Model comparisons for predicting NSLN metastasis among SLN+ patients (Bay Area SLN Database).</p></caption><table frame="hsides" rules="groups"><thead><tr><td></td><td align="center"><bold>For all SLN+ Pts (n = 285)</bold></td><td align="center"><bold>For pts with known angiolymphatic invasion status (n = 213)</bold></td><td align="center"><bold>For pts with known angiolymphatic invasion and ER status (n = 171)</bold></td></tr></thead><tbody><tr><td align="left"><bold>RP-ROC with 10-fold cross validation</bold></td><td></td><td></td><td></td></tr><tr><td align="left">Sensitivity (%)</td><td></td><td align="center">78.8</td><td align="center">83.2</td></tr><tr><td align="left">Specificity (%)</td><td></td><td align="center">75.5</td><td align="center">78.1</td></tr><tr><td align="left">Diagnostic Accuracy by AUC (%)</td><td></td><td align="center"><bold>76.7</bold></td><td align="center"><bold>80.2</bold></td></tr><tr><td></td><td></td><td></td><td></td></tr><tr><td align="left"><bold>Boosted CART with 10-fold cross validation</bold></td><td></td><td></td><td></td></tr><tr><td align="left">Sensitivity (%)</td><td align="center">78.2</td><td align="center">87.9</td><td align="center">89.0</td></tr><tr><td align="left">Specificity (%)</td><td align="center">62.0</td><td align="center">71.4</td><td align="center">74.7</td></tr><tr><td align="left">Diagnostic Accuracy by AUC (%)</td><td align="center"><bold>67.7</bold></td><td align="center"><bold>77.5</bold></td><td align="center"><bold>80.3</bold></td></tr><tr><td></td><td></td><td></td><td></td></tr><tr><td align="left"><bold>Multivariable logistic regression with 10-fold cross validation</bold></td><td></td><td></td><td></td></tr><tr><td align="left">Sensitivity (%)</td><td></td><td align="center">78.0</td><td align="center">78.9</td></tr><tr><td align="left">Specificity (%)</td><td></td><td align="center">86.2</td><td align="center">88.3</td></tr><tr><td align="left">Diagnostic Accuracy by AUC (%)</td><td></td><td align="center"><bold>83.3</bold></td><td align="center"><bold>84.9</bold></td></tr><tr><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td></tr><tr><td align="left"><bold>MSKCC Breast Cancer Nomogram</bold><sup>a</sup></td><td></td><td></td><td></td></tr><tr><td align="left">Diagnsotic Accuracy by AUC (%)</td><td></td><td></td><td align="center"><bold>76.7</bold></td></tr></tbody></table><table-wrap-foot><p><sup>a </sup>Memorial Sloan-Kettering Cancer Center Breast Cancer Nomogram for Prediction of Axillary Lymph Node Metastasis applied to SLN+ pts in Stanford dataset who have complete data for all Nomogram variables</p></table-wrap-foot></table-wrap><p>Finally, the MLR and Nomogram were applied to a database of 77 patients who received ALND for positive SLNs at Northwestern University (Additional file <xref ref-type="supplementary-material" rid="S6">6</xref>). The SLN metastases in this dataset were identified by H&E stain without IHC. Among the 77 SLN positive patients, 61% had T1 tumors, 36% had T2 tumors, and 2.6% had T3 tumors. Angiolymphatic invasion was present in 68% of patients' tumors, and the SLN metastases in the Northwestern dataset were predominantly of large tumor burden with 56% having macrometastasis. NSLN metastases were present in 24 patients (31%). This is in contrast to the Bay Area SLN dataset with 55% T1 tumors, 38% T2 tumors, and 7% T3 tumors; 45% with angiolymphatic invasion (when angiolymphatic invasion status was known); 7% with macrometastasis; and 35% NSLN metastases (Table <xref ref-type="table" rid="T1">1</xref> and Figure <xref ref-type="fig" rid="F1">1</xref>). Although the Northwestern tumors were somewhat smaller, the higher percentage of angiolymphatic invasion and SLN macrometastases suggest more biologically aggressive disease in their dataset, yet they had a slightly lower percentage of NSLN metastasis.</p><p>Both the MLR model and the Nomogram performed less well when applied to the Northwestern dataset; however, the MLR model was supported with an AUC of 77% (95% confidence interval 0.67–0.80). This is superior to the performance of the Nomogram among this population, 62% (95% confidence interval 0.55–0.68) (Figure <xref ref-type="fig" rid="F4">4</xref>).</p><fig position="float" id="F4"><label>Figure 4</label><caption><p><bold>ROC curves for MLR informed by CART calculation in blue, AUC = 0.74, and Nomogram in green, AUC = 0.62, when applied to the Northwestern test set (n = 77).</bold> 24 patients had NSLN metastasis in this dataset.</p></caption><graphic xlink:href="1471-2407-8-66-4"></graphic></fig></sec><sec><title>Discussion</title><p>Sentinel lymph node biopsy is a major advance in the treatment of women with breast cancer [<xref ref-type="bibr" rid="B35">35</xref>]. If no SLN metastases are identified, the likelihood of additional NSLN involvement is 9.8% in our series. Though above the goal false-negative rate proposed by the American Society of Breast Surgeons, this is comparable to that reported in NSABP-32 and recently by both Lyman and Veronesi ranging 9.7%, 8.4%, and 8.8% respectively [<xref ref-type="bibr" rid="B1">1</xref>,<xref ref-type="bibr" rid="B36">36</xref>,<xref ref-type="bibr" rid="B37">37</xref>]. Of our patients with positive SLNs, the majority presented with micrometastasis, 70%, or isolated tumor cells, 22%. Thus, our population contains a predominance of limited SLN disease burden relative to prior reports, including van Rijk's reported rate of 23% for micrometastasis and 16% for isolated tumor cells [<xref ref-type="bibr" rid="B38">38</xref>]. This may be important as suggested by Alran et al. who showed lower performance of the Nomogram in patients with only micrometastases [<xref ref-type="bibr" rid="B19">19</xref>]. Despite the seemingly low sentinel node tumor burden, 35% had NSLN metastases upon completion ALND. Unfortunately, no combination of clinical and/or pathologic characteristics enabled identification of all SLN-positive patients at risk for NSLN metastases. Although SLN-positive patients will receive systemic chemotherapy and/or hormone therapy, it is unknown whether occult NSLN metastases are eradicated by adjuvant treatment. Until results of large prospective clinical trials can demonstrate no long-term increase in mortality from omitting ALND in the setting of systemic therapy, prophylactic ALND for patients with tumor-involved SLNs, including those with and without NSLN involvement, remains standard surgical care [<xref ref-type="bibr" rid="B39">39</xref>-<xref ref-type="bibr" rid="B43">43</xref>]. However, in practice, it is the patient and her physician who decide whether or not a completion axillary dissection is performed. This decision may be informed using online calculators such as the Nomogram and the one presented here.</p><p>Based on a multi-institutional sample set larger than most prior studies, we found that univariate predictors of NSLN status include tumor size (in cm and by AJCC T size classification), tumor grade, hormone receptor status (ER and PR), angiolymphatic invasion, size of SLN metastasis, and whether nodal tumor involvement is identified by H&E. By multivariate analyses, tumor size, angiolymphatic invasion, size of SLN metastases, and products of these variables predict NSLN tumor involvement. Others have also discovered the predictive strength of each of the three simple characteristics [<xref ref-type="bibr" rid="B5">5</xref>,<xref ref-type="bibr" rid="B6">6</xref>,<xref ref-type="bibr" rid="B10">10</xref>,<xref ref-type="bibr" rid="B44">44</xref>-<xref ref-type="bibr" rid="B50">50</xref>], although here we confirm their collective power in a unique way. Additionally, we found that angiolymphatic invasion is as strong a predictor of NSLN metastasis as is size of SLN metastasis.</p><p>For women with isolated tumor cells in the SLN, we found a 4.7% chance of NSLN involvement, similar to Calhoun and Giuliano's reported NSLN-involvement rate of 4.9% for the same subset of patients, and comparable to or lower than that found previously, 10–15% [<xref ref-type="bibr" rid="B47">47</xref>,<xref ref-type="bibr" rid="B51">51</xref>,<xref ref-type="bibr" rid="B52">52</xref>]. The benefits of no further axillary dissection must be weighed against the risk of harboring axillary metastasis that may potentially seed occult metastatic disease. Clinical context, with consideration of a patient's expected life-span and associated health problems, may impact the definition of a "minimal acceptable risk." Recommendations for clinical practice are difficult because the risk of NSLN metastasis in SLN-positive patients with isolated tumor cells is comparable or lower than the risk of NSLN metastases in patients without SLN metastases (9.8% in our study) [<xref ref-type="bibr" rid="B53">53</xref>]. These issues are being studied in large-scale prospective clinical trials [<xref ref-type="bibr" rid="B40">40</xref>,<xref ref-type="bibr" rid="B41">41</xref>]. Future molecular technologies may also provide guidance [<xref ref-type="bibr" rid="B54">54</xref>,<xref ref-type="bibr" rid="B55">55</xref>].</p><p>Our goal was to identify patients with tumor-free NSLNs who, with near certainty, may be spared completion ALND. Using multivariate tree-based modeling by RP-ROC, boosted CART, and MLR informed by CART, we identified tumor size, angiolymphatic invasion, and size of SLN metastasis as characteristics that optimized stratification of NSLN status. These refined, statistical analyses demonstrated a highly synergistic interaction between size of SLN metastasis and angiolymphatic invasion on risk of NSLN metastasis. Our models (Figures <xref ref-type="fig" rid="F2">2</xref>, Additional files <xref ref-type="supplementary-material" rid="S4">4</xref> and <xref ref-type="supplementary-material" rid="S5">5</xref>) stratified patients with tumor-involved SLNs into four risk groups for having NSLN metastasis: low risk (10% or less), moderate risk (30–45%), high risk (about 60%), and very high risk (greater than 90%).</p><p>MLR modeling of NSLN status that was informed by CART in its selection of predictors provided the most accurate cross-validated technique for predicting NSLN metastases for patients with known angiolymphatic invasive status, with accuracy superior to boosted CART, RP-ROC, and the Memorial Sloan-Kettering Breast Cancer Nomogram. When applied to the Bay Area SLN Database, the Nomogram had an AUC of 77%. This compares with the accuracy of the Nomogram for the original MSKCC population (76%) and for a prospective cohort at MSKCC (77%) [<xref ref-type="bibr" rid="B13">13</xref>]. Seven subsequent studies have tested the Nomogram and show an accuracy of 63% to 86%, though as low as 54% when applied only to patients with SLN micrometastases [<xref ref-type="bibr" rid="B14">14</xref>,<xref ref-type="bibr" rid="B17">17</xref>-<xref ref-type="bibr" rid="B19">19</xref>,<xref ref-type="bibr" rid="B56">56</xref>-<xref ref-type="bibr" rid="B60">60</xref>]. In contrast, our MLR informed by CART model performed equally well among patients with isolated tumor cells, micrometastases, or macrometastases. Relative importance of size of SLN metastasis in the Nomogram is determined by method of detection including IHC, serial H&E, routine analysis, versus frozen section (among the subset for which this is performed) [<xref ref-type="bibr" rid="B13">13</xref>]. The improved predictive accuracy of the MLR model informed by CART, particularly among patients with isolated tumor cells or micrometastasis, may be due to the relative weight ascribed to the specific size of SLN metastasis in our model. Application of our MLR model to other patient populations is required to validate its performance.</p><p>Considering the risk of potential bias due to low sentinel node tumor burden in our dataset, we applied both our model and the MSKCC Nomogram to an independent dataset of 77 SLN positive patients who underwent completion ALND. These cases were not identified by IHC and this dataset contained cases with a much larger tumor burden: 56% of cases contained macrometastasis in the SLN compared to 7% of the Bay Area SLN dataset. Again, the MLR model showed superior performance to the Nomogram. However, the performance of both models decreased compared to the Bay Area SLN Database: the Stanford Online Calculator generated an AUC of 0.74, or 74%, and the Nomogram generated an AUC of 0.62, or 62%. This raises concern regarding the generalizable nature of any model. An underlying reason why neither model performed as well as anticipated is that when a model is developed based on data from one group of patients, and the model is subsequently applied to data from a different group of patients, performance is generally diminished [<xref ref-type="bibr" rid="B24">24</xref>,<xref ref-type="bibr" rid="B27">27</xref>]. This is due to differences in the distributions of predictive features and differences in the synergistic impact (interactions) between and among these features in different groups of patients. Thus a model developed in one group of patients would not be expected to perform as well for a different group of patients, even if the performance of the model was validated internally (cross-validated) on the original group. Table <xref ref-type="table" rid="T1">1</xref> and Additional file <xref ref-type="supplementary-material" rid="S6">6</xref> shows differences in the distribution of the three variables in our model – tumor size, size of SLN metastasis, and angiolymphatic invasion – between patients in the Bay Area SLN Database and Northwestern series. As we would also expect the interactions of these variables to be different for both groups, we believe these factors in aggregate may be responsible for our findings.</p><p>The predictive accuracy of the Nomogram requires assessment of eight tumor characteristics [<xref ref-type="bibr" rid="B13">13</xref>,<xref ref-type="bibr" rid="B14">14</xref>,<xref ref-type="bibr" rid="B17">17</xref>,<xref ref-type="bibr" rid="B18">18</xref>,<xref ref-type="bibr" rid="B56">56</xref>]. A hazard of multi-variable modeling is that its overall accuracy is dependent upon the accuracy and precision with which each individual variable is determined. Our MLR confirmed the importance of two composite variables from only three tumor characteristics: 1) the size of SLN metastasis when angiolymphatic invasion is present, and 2) tumor size times the square of the size of SLN metastasis. The first composite variable reflects the synergism between angiolymphatic invasion and size of SLN metastasis; the second involves tumor burden. Using these two composite variables, AUC is 83% or 85% compared to the Nomogram's AUC of 77% that relies on eight variables. By using statistical methods which allow assessment of the variable-variable interactions we demonstrate superior accuracy with fewer required variables. Our model is the first proposed which emphasizes the synergistic interactions among patient characteristics. By reducing the required variables, we are hopeful the MLR model may be applied a larger population of patients, without excluding those with incomplete, unavailable, or pending pathologic data.</p><p>Missing pathologic data is problematic for breast cancer patients nationwide. Though the generalizability of our model may have benefited from the diverse population represented, obtaining complete clinicopathologic information was partially limited by enrollment across 16 institutions during the years of our study, 1996–2002. Approximately 25% of our 285 SLN-positive patients had no histologic analysis for angiolymphatic invasion. Of the 213 patients with angiolymphatic data present, another 19.7% had no ER status performed or recorded. This is comparable to the 17.1% of invasive breast cancer patients without recorded ER status in 13 registries of the national Surveillance, Epidemiology, and End Results (SEER) database from 1999–2003 [<xref ref-type="bibr" rid="B61">61</xref>] (unpublished data, Jeffrey lab); presence of angiolymphatic invasion status was not requested by SEER. Thus, we analyzed our data using three patient groups: the entire SLN-positive dataset of 285 patients; 213 SLN-positive patients who had complete information on angiolymphatic invasion; and 171 SLN-positive patients who had complete information on angiolymphatic invasion and ER status. Even applying the smallest dataset, more SLN-positive patients are analyzed than in most other published studies.</p><p>Though not directly compared among identical patient populations, the AUC of our model is also superior to that of M.D. Anderson Cancer Center scoring system (70%) and the Hôpital Tenon scoring system derived in Paris, France (68%), as recently reported by Dauphine <italic>et al. </italic>[<xref ref-type="bibr" rid="B59">59</xref>,<xref ref-type="bibr" rid="B62">62</xref>,<xref ref-type="bibr" rid="B63">63</xref>]. Calculations using our MLR model are easily done over the internet with the Stanford Online Calculator [<xref ref-type="bibr" rid="B64">64</xref>]. We encourage others to access and test our model and directly compare it with other models for evaluating risk of NSLN metastasis.</p><p>Although no modeling technique has been able to identify patients without any risk of NSLN metastasis, Park <italic>et al</italic>. recently argued that ALND may be reasonably eliminated among patients with approximately 9% or less predicted risk of NSLN involvement [<xref ref-type="bibr" rid="B16">16</xref>]. A low risk subset of 287 patients with SLN metastasis were followed in a non-randomized study with a 2% observed rate of local recurrence. This recommendation, however, is limited by a follow-up of only 23 months. We expect that data from two large prospective clinical studies, NSABP-32 and American College of Surgeons trial Z0011, will more definitively resolve questions regarding the optimal surgical management of SLN-positive patients [<xref ref-type="bibr" rid="B40">40</xref>,<xref ref-type="bibr" rid="B41">41</xref>]. In the meantime, we hope that our calculator may provide further guidance for risk evaluation.</p></sec><sec><title>Conclusion</title><p>Fewer than half of women undergoing completion axillary lymph node dissection (ALND) for breast cancer will have non-sentinel node (NSLN) metastasis. We present a new model and the Stanford Online Calculator developed from a Northern California and Oregon database with superior accuracy and simplicity (three versus eight required patient variables) compared to the Memorial Sloan-Kettering Breast Cancer Nomogram for our dataset and another independent dataset. We hope that other institutions will test our model using their datasets, which will contain different patient demographics, to validate its accuracy and to refine in which populations it may be best used. Further investigation of predictive models to stratify risk of non-sentinel lymph node metastasis will better define their role in guiding clinical decision-making, while we await the results of larger randomized trials.</p></sec><sec><title>Abbreviations</title><p>ALND, axillary lymph node dissection; LN, lymph node; SLN, sentinel lymph node; NSLN, non-sentinel lymph node; RP-ROC, recursive partitioning with receiver operating characteristic curves; CART, boosted classification and regression trees; MLR, multivariate logistic regression; MSKCC, Memorial Sloan-Kettering Cancer Center; Nomogram, MSKCC Breast Cancer Nomogram; ER, estrogen receptor; PR, progesterone receptor; AUC, area under ROC curve; LVI, lymphovascular invasion; H&E, hematoxylin and eosin; IHC, immunohistochemistry; ITC, isolated tumor cells.</p></sec><sec><title>Competing interests</title><p>Richard A. Olshen is one of the original developers of CART<sup>®</sup>, the software for which is distributed by Salford Systems of San Diego, CA. The remaining authors have no competing interests to declare.</p></sec><sec><title>Authors' contributions</title><p>HEK, RAO, and SSJ conceived the study, performed data analysis, and drafted the manuscript with critical input from RWC, FMD, WHG, DLJ, RVR, FES, and ILW. LB, RWC, FMD, JJD, WHG, SSJ, DLJ, VJP, and FES developed the Bay Area SLN Study Database and assisted in patient accrual, clinical data acquisition, or data review. RVR processed and analyzed sentinel node tissue blocks. DJW and SH wrote the software for the Stanford Online Calculator and created the website. HRB obtained IRB approval and compiled all Northwestern SLN test data under the guidance of NMH. All authors read and approved the final manuscript.</p></sec><sec><title>Pre-publication history</title><p>The pre-publication history for this paper can be accessed here:</p><p><ext-link ext-link-type="uri" xlink:href="http://www.biomedcentral.com/1471-2407/8/66/prepub"></ext-link></p></sec><sec sec-type="supplementary-material"><title>Supplementary Material</title><supplementary-material content-type="local-data" id="S1"><caption><title>Additional file 1</title><p>Schematic of patients accrued to Bay Area SLN Database. An overview of the entire Bay Area SLN Database and exclusion criteria for this study.</p></caption><media xlink:href="1471-2407-8-66-S1.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media></supplementary-material><supplementary-material content-type="local-data" id="S2"><caption><title>Additional file 2</title><p>Patient, primary tumor, and lymph node characteristics among SLN-negative and SLN-positive patients from the Bay Area SLN Database. This table describes the demographics of the patient, primary tumor, and lymph node characteristics among SLN-negative and SLN-positive patients from the Bay Area SLN Database.</p></caption><media xlink:href="1471-2407-8-66-S2.doc" mimetype="application" mime-subtype="msword"><caption><p>Click here for file</p></caption></media></supplementary-material><supplementary-material content-type="local-data" id="S3"><caption><title>Additional file 3</title><p>The relationship of angiolymphatic invasion and size of SLN metastasis to tumor size (Bay Area SLN Database). This table shows A. the occurrence of angiolymphatic invasion with increasing tumor size for SLN-negative and SLN-positive patients, and B. size of SLN metastasis with increasing tumor size (Bay Area SLN Database).</p></caption><media xlink:href="1471-2407-8-66-S3.doc" mimetype="application" mime-subtype="msword"><caption><p>Click here for file</p></caption></media></supplementary-material><supplementary-material content-type="local-data" id="S4"><caption><title>Additional file 4</title><p>CART decision tree for patients with complete data on angiolymphatic invasion status, n = 213 (Bay Area SLN Database). The figure shows the CART decision tree for 213 patients with complete data on angiolymphatic invasion status.</p></caption><media xlink:href="1471-2407-8-66-S4.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media></supplementary-material><supplementary-material content-type="local-data" id="S5"><caption><title>Additional file 5</title><p>CART decision tree for patients with complete data on angiolymphatic invasion status and ER status, n = 171 (Bay Area SLN Database). The figure shows the CART decision tree for patients with complete data on angiolymphatic invasion status and ER status.</p></caption><media xlink:href="1471-2407-8-66-S5.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media></supplementary-material><supplementary-material content-type="local-data" id="S6"><caption><title>Additional file 6</title><p>Patient, primary tumor, and lymph node characteristics among SLN-positive and SLN-negative patients from the Northwestern dataset. The table describes the demographics of the patient, primary tumor, and lymph node characteristics among SLN-negative and SLN-positive patients from the Northwestern dataset.</p></caption><media xlink:href="1471-2407-8-66-S6.doc" mimetype="application" mime-subtype="msword"><caption><p>Click here for file</p></caption></media></supplementary-material></sec></body><back><ack><sec><title>Acknowledgements</title><p>The authors thank Mme. Sophie DuLac for her generous support of the Bay Area SLN Study. RAO was supported in part by NIH grant 2-R01-EB-002784-30. We are indebted to Elizabeth Spence, Sunita Jones, Ph.D., Maureen Chang, and Susan Overholser for their skilled work in database management. We thank Bonnie Chung, Balasubramanian Narasimhan, and Adam Kapelner for their assistance in creating and improving the Web site and Stanford Online Calculator. We also thank David Siegmund, Ph.D., of Stanford's Department of Statistics for assistance in the development and initial analysis of the Bay Area Sentinel Node Database, as well as Dr. Don Goffinet of Stanford's Department of Radiation Oncology and current and former members of Stanford's Division of Nuclear Medicine, Drs. Michael Goris, Ross McDougall, William Pace, and H. William Strauss for facilitating this investigation.</p><p>We acknowledge all physicians and institutions participating in the <italic>Bay Area SLN Study </italic>including: <bold>Stanford University Medical Center, Stanford, CA</bold>: Stefanie Jeffrey, MD (Principal Investigator); James Badger, MD, Robyn Birdwell, MD, Martin Bronk, MD, Robert Carlson, MD, Frederick Dirbas, MD, Jocelyn Dunn, MD, Don Goffinet, MD, David Gregg, MD, Debra Ikeda, MD, Denise Johnson, MD, Gail Lebovic, MD, Mary Ellen Mahoney, MD, Richard Olshen, PhD, William Pace, MD, Robert Rouse, MD, Melanie Smitt, MD, Lynn Smolik, MD, Frank Stockdale, MD, H. William Strauss, MD, and Ward Trueblood, MD; <bold>Alta Bates Summit Medical Center, Berkeley, CA</bold>: Charles Jenkins, MD, and Lisa Bailey, MD; <bold>California Pacific Medical Center, SF, CA</bold>: William Goodson III, MD; <bold>Doctor's Medical Center, San Pablo, CA</bold>: Stuart J Gourlay, MD; <bold>Dominican Santa Cruz Hospital, Santa Cruz, CA</bold>: David Albritton, MD, and David Rose, MD; <bold>El Camino Hospital/El Camino Surgery Center, Mountain View, CA</bold>: Alfred Butner, MD; <bold>Enloe Hospital, Chico, CA</bold>: William Battinich, MD, F. David Collins, MD, and Eugene Cleek, MD; <bold>Mercy Medical Center, Redding, CA</bold>: Vicki Philben, MD; <bold>Mt. Diablo Hospital, Concord, CA</bold>: Burton Baker, MD, and Don Beerline, MD; <bold>Queen of the Valley Hospital, Napa, CA</bold>: Kristen Engle, MD, Leland Raymond, MD, and Wendell Wenneker, MD; <bold>San Ramon Regional Center, San Ramon, CA</bold>: Mary Estakhri, MD, and Michael Wynn, MD; <bold>Seton Hospital, Daly City, CA</bold>: Henry Chin, MD, and Dorothy McNoble, MD; <bold>St. Josephs Hospital, Stockton, CA</bold>: Dean Sloan, MD; <bold>The Sutter Cancer Center, Sacramento, CA</bold>: Vincent Caggiano, MD, Joyce Eaker, MD, Jay Owens, MD, and Mark Roberts, MD; <bold>Sutter Roseville Medical Center, Roseville, CA</bold>: Peter Krone, MD, Alan McNabb, MD, and Frederick Weiland, MD; <bold>Willamette Valley Medical Center, McMinnville, OR</bold>: Harold Hoover, MD, Harry McCulley, MD, and Erik Swensson, MD.</p></sec></ack><ref-list><ref id="B1"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lyman</surname><given-names>GH</given-names></name><name><surname>Giuliano</surname><given-names>AE</given-names></name><name><surname>Somerfield</surname><given-names>MR</given-names></name><name><surname>Benson</surname><given-names>AB</given-names><suffix>3rd</suffix></name><name><surname>Bodurka</surname><given-names>DC</given-names></name><name><surname>Burstein</surname><given-names>HJ</given-names></name><name><surname>Cochran</surname><given-names>AJ</given-names></name><name><surname>Cody</surname><given-names>HS</given-names><suffix>3rd</suffix></name><name><surname>Edge</surname><given-names>SB</given-names></name><name><surname>Galper</surname><given-names>S</given-names></name><name><surname>Hayman</surname><given-names>JA</given-names></name><name><surname>Kim</surname><given-names>TY</given-names></name><name><surname>Perkins</surname><given-names>CL</given-names></name><name><surname>Podoloff</surname><given-names>DA</given-names></name><name><surname>Sivasubramaniam</surname><given-names>VH</given-names></name><name><surname>Turner</surname><given-names>RR</given-names></name><name><surname>Wahl</surname><given-names>R</given-names></name><name><surname>Weaver</surname><given-names>DL</given-names></name><name><surname>Wolff</surname><given-names>AC</given-names></name><name><surname>Winer</surname><given-names>EP</given-names></name></person-group><article-title>American Society of Clinical Oncology guideline recommendations for sentinel lymph node biopsy in early-stage breast cancer</article-title><source>J Clin Oncol</source><year>2005</year><volume>23</volume><fpage>7703</fpage><lpage>7720</lpage><pub-id pub-id-type="pmid">16157938</pub-id><pub-id pub-id-type="doi">10.1200/JCO.2005.08.001</pub-id></citation></ref><ref id="B2"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rubio</surname><given-names>IT</given-names></name><name><surname>Korourian</surname><given-names>S</given-names></name><name><surname>Cowan</surname><given-names>C</given-names></name><name><surname>Krag</surname><given-names>DN</given-names></name><name><surname>Colvert</surname><given-names>M</given-names></name><name><surname>Klimberg</surname><given-names>VS</given-names></name></person-group><article-title>Sentinel lymph node biopsy for staging breast cancer</article-title><source>Am J Surg</source><year>1998</year><volume>176</volume><fpage>532</fpage><lpage>537</lpage><pub-id pub-id-type="pmid">9926785</pub-id><pub-id pub-id-type="doi">10.1016/S0002-9610(98)00264-5</pub-id></citation></ref><ref id="B3"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Carlson</surname><given-names>RW</given-names></name><name><surname>McCormick</surname><given-names>B</given-names></name></person-group><article-title>Update: NCCN breast cancer Clinical Practice Guidelines</article-title><source>J Natl Compr Canc Netw</source><year>2005</year><volume>3 Suppl 1</volume><fpage>S7</fpage><lpage>11</lpage><pub-id pub-id-type="pmid">16280118</pub-id></citation></ref><ref id="B4"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Sosa</surname><given-names>JA</given-names></name><name><surname>Diener-West</surname><given-names>M</given-names></name><name><surname>Gusev</surname><given-names>Y</given-names></name><name><surname>Choti</surname><given-names>MA</given-names></name><name><surname>Lange</surname><given-names>JR</given-names></name><name><surname>Dooley</surname><given-names>WC</given-names></name><name><surname>Zeiger</surname><given-names>MA</given-names></name></person-group><article-title>Association between extent of axillary lymph node dissection and survival in patients with stage I breast cancer</article-title><source>Ann Surg Oncol</source><year>1998</year><volume>5</volume><fpage>140</fpage><lpage>149</lpage><pub-id pub-id-type="pmid">9527267</pub-id><pub-id pub-id-type="doi">10.1007/BF02303847</pub-id></citation></ref><ref id="B5"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Chu</surname><given-names>KU</given-names></name><name><surname>Turner</surname><given-names>RR</given-names></name><name><surname>Hansen</surname><given-names>NM</given-names></name><name><surname>Brennan</surname><given-names>MB</given-names></name><name><surname>Bilchik</surname><given-names>A</given-names></name><name><surname>Giuliano</surname><given-names>AE</given-names></name></person-group><article-title>Do all patients with sentinel node metastasis from breast carcinoma need complete axillary node dissection?</article-title><source>Ann Surg</source><year>1999</year><volume>229</volume><fpage>536</fpage><lpage>541</lpage><pub-id pub-id-type="pmid">10203087</pub-id><pub-id pub-id-type="doi">10.1097/00000658-199904000-00013</pub-id></citation></ref><ref id="B6"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Turner</surname><given-names>RR</given-names></name><name><surname>Chu</surname><given-names>KU</given-names></name><name><surname>Qi</surname><given-names>K</given-names></name><name><surname>Botnick</surname><given-names>LE</given-names></name><name><surname>Hansen</surname><given-names>NM</given-names></name><name><surname>Glass</surname><given-names>EC</given-names></name><name><surname>Giuliano</surname><given-names>AE</given-names></name></person-group><article-title>Pathologic features associated with nonsentinel lymph node metastases in patients with metastatic breast carcinoma in a sentinel lymph node</article-title><source>Cancer</source><year>2000</year><volume>89</volume><fpage>574</fpage><lpage>581</lpage><pub-id pub-id-type="pmid">10931456</pub-id><pub-id pub-id-type="doi">10.1002/1097-0142(20000801)89:3<574::AID-CNCR12>3.0.CO;2-Y</pub-id></citation></ref><ref id="B7"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rietman</surname><given-names>JS</given-names></name><name><surname>Dijkstra</surname><given-names>PU</given-names></name><name><surname>Geertzen</surname><given-names>JH</given-names></name><name><surname>Baas</surname><given-names>P</given-names></name><name><surname>De Vries</surname><given-names>J</given-names></name><name><surname>Dolsma</surname><given-names>W</given-names></name><name><surname>Groothoff</surname><given-names>JW</given-names></name><name><surname>Eisma</surname><given-names>WH</given-names></name><name><surname>Hoekstra</surname><given-names>HJ</given-names></name></person-group><article-title>Short-term morbidity of the upper limb after sentinel lymph node biopsy or axillary lymph node dissection for Stage I or II breast carcinoma</article-title><source>Cancer</source><year>2003</year><volume>98</volume><fpage>690</fpage><lpage>696</lpage><pub-id pub-id-type="pmid">12910511</pub-id><pub-id pub-id-type="doi">10.1002/cncr.11545</pub-id></citation></ref><ref id="B8"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hack</surname><given-names>TF</given-names></name><name><surname>Cohen</surname><given-names>L</given-names></name><name><surname>Katz</surname><given-names>J</given-names></name><name><surname>Robson</surname><given-names>LS</given-names></name><name><surname>Goss</surname><given-names>P</given-names></name></person-group><article-title>Physical and psychological morbidity after axillary lymph node dissection for breast cancer</article-title><source>J Clin Oncol</source><year>1999</year><volume>17</volume><fpage>143</fpage><lpage>149</lpage><pub-id pub-id-type="pmid">10458227</pub-id></citation></ref><ref id="B9"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Mansel</surname><given-names>RE</given-names></name><name><surname>Fallowfield</surname><given-names>L</given-names></name><name><surname>Kissin</surname><given-names>M</given-names></name><name><surname>Goyal</surname><given-names>A</given-names></name><name><surname>Newcombe</surname><given-names>RG</given-names></name><name><surname>Dixon</surname><given-names>JM</given-names></name><name><surname>Yiangou</surname><given-names>C</given-names></name><name><surname>Horgan</surname><given-names>K</given-names></name><name><surname>Bundred</surname><given-names>N</given-names></name><name><surname>Monypenny</surname><given-names>I</given-names></name><name><surname>England</surname><given-names>D</given-names></name><name><surname>Sibbering</surname><given-names>M</given-names></name><name><surname>Abdullah</surname><given-names>TI</given-names></name><name><surname>Barr</surname><given-names>L</given-names></name><name><surname>Chetty</surname><given-names>U</given-names></name><name><surname>Sinnett</surname><given-names>DH</given-names></name><name><surname>Fleissig</surname><given-names>A</given-names></name><name><surname>Clarke</surname><given-names>D</given-names></name><name><surname>Ell</surname><given-names>PJ</given-names></name></person-group><article-title>Randomized multicenter trial of sentinel node biopsy versus standard axillary treatment in operable breast cancer: the ALMANAC Trial</article-title><source>J Natl Cancer Inst</source><year>2006</year><volume>98</volume><fpage>599</fpage><lpage>609</lpage><pub-id pub-id-type="pmid">16670385</pub-id></citation></ref><ref id="B10"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Reynolds</surname><given-names>C</given-names></name><name><surname>Mick</surname><given-names>R</given-names></name><name><surname>Donohue</surname><given-names>JH</given-names></name><name><surname>Grant</surname><given-names>CS</given-names></name><name><surname>Farley</surname><given-names>DR</given-names></name><name><surname>Callans</surname><given-names>LS</given-names></name><name><surname>Orel</surname><given-names>SG</given-names></name><name><surname>Keeney</surname><given-names>GL</given-names></name><name><surname>Lawton</surname><given-names>TJ</given-names></name><name><surname>Czerniecki</surname><given-names>BJ</given-names></name></person-group><article-title>Sentinel lymph node biopsy with metastasis: can axillary dissection be avoided in some patients with breast cancer?</article-title><source>J Clin Oncol</source><year>1999</year><volume>17</volume><fpage>1720</fpage><lpage>1726</lpage><pub-id pub-id-type="pmid">10561208</pub-id></citation></ref><ref id="B11"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Abdessalam</surname><given-names>SF</given-names></name><name><surname>Zervos</surname><given-names>EE</given-names></name><name><surname>Prasad</surname><given-names>M</given-names></name><name><surname>Farrar</surname><given-names>WB</given-names></name><name><surname>Yee</surname><given-names>LD</given-names></name><name><surname>Walker</surname><given-names>MJ</given-names></name><name><surname>Carson</surname><given-names>WB</given-names></name><name><surname>Burak</surname><given-names>WE</given-names><suffix>Jr.</suffix></name></person-group><article-title>Predictors of positive axillary lymph nodes after sentinel lymph node biopsy in breast cancer</article-title><source>Am J Surg</source><year>2001</year><volume>182</volume><fpage>316</fpage><lpage>320</lpage><pub-id pub-id-type="pmid">11720662</pub-id><pub-id pub-id-type="doi">10.1016/S0002-9610(01)00719-X</pub-id></citation></ref><ref id="B12"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rahusen</surname><given-names>FD</given-names></name><name><surname>Torrenga</surname><given-names>H</given-names></name><name><surname>van Diest</surname><given-names>PJ</given-names></name><name><surname>Pijpers</surname><given-names>R</given-names></name><name><surname>van der Wall</surname><given-names>E</given-names></name><name><surname>Licht</surname><given-names>J</given-names></name><name><surname>Meijer</surname><given-names>S</given-names></name></person-group><article-title>Predictive factors for metastatic involvement of nonsentinel nodes in patients with breast cancer</article-title><source>Arch Surg</source><year>2001</year><volume>136</volume><fpage>1059</fpage><lpage>1063</lpage><pub-id pub-id-type="pmid">11529831</pub-id><pub-id pub-id-type="doi">10.1001/archsurg.136.9.1059</pub-id></citation></ref><ref id="B13"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Van Zee</surname><given-names>KJ</given-names></name><name><surname>Manasseh</surname><given-names>DM</given-names></name><name><surname>Bevilacqua</surname><given-names>JL</given-names></name><name><surname>Boolbol</surname><given-names>SK</given-names></name><name><surname>Fey</surname><given-names>JV</given-names></name><name><surname>Tan</surname><given-names>LK</given-names></name><name><surname>Borgen</surname><given-names>PI</given-names></name><name><surname>Cody</surname><given-names>HS</given-names><suffix>3rd</suffix></name><name><surname>Kattan</surname><given-names>MW</given-names></name></person-group><article-title>A nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy</article-title><source>Ann Surg Oncol</source><year>2003</year><volume>10</volume><fpage>1140</fpage><lpage>1151</lpage><pub-id pub-id-type="pmid">14654469</pub-id><pub-id pub-id-type="doi">10.1245/ASO.2003.03.015</pub-id></citation></ref><ref id="B14"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Smidt</surname><given-names>ML</given-names></name><name><surname>Kuster</surname><given-names>DM</given-names></name><name><surname>van der Wilt</surname><given-names>GJ</given-names></name><name><surname>Thunnissen</surname><given-names>FB</given-names></name><name><surname>Van Zee</surname><given-names>KJ</given-names></name><name><surname>Strobbe</surname><given-names>LJ</given-names></name></person-group><article-title>Can the Memorial Sloan-Kettering Cancer Center Nomogram predict the likelihood of nonsentinel lymph node metastases in breast cancer patients in the Netherlands?</article-title><source>Ann Surg Oncol</source><year>2005</year><volume>12</volume><fpage>1066</fpage><lpage>1072</lpage><pub-id pub-id-type="pmid">16244802</pub-id><pub-id pub-id-type="doi">10.1245/ASO.2005.07.022</pub-id></citation></ref><ref id="B15"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Specht</surname><given-names>MC</given-names></name><name><surname>Kattan</surname><given-names>MW</given-names></name><name><surname>Gonen</surname><given-names>M</given-names></name><name><surname>Fey</surname><given-names>J</given-names></name><name><surname>Van Zee</surname><given-names>KJ</given-names></name></person-group><article-title>Predicting nonsentinel node status after positive sentinel lymph biopsy for breast cancer: clinicians versus nomogram</article-title><source>Ann Surg Oncol</source><year>2005</year><volume>12</volume><fpage>654</fpage><lpage>659</lpage><pub-id pub-id-type="pmid">16021535</pub-id><pub-id pub-id-type="doi">10.1245/ASO.2005.06.037</pub-id></citation></ref><ref id="B16"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Park</surname><given-names>J</given-names></name><name><surname>Fey</surname><given-names>JV</given-names></name><name><surname>Naik</surname><given-names>AM</given-names></name><name><surname>Borgen</surname><given-names>PI</given-names></name><name><surname>Van Zee</surname><given-names>KJ</given-names></name><name><surname>Cody</surname><given-names>HS</given-names><suffix>3rd</suffix></name></person-group><article-title>A declining rate of completion axillary dissection in sentinel lymph node-positive breast cancer patients is associated with the use of a multivariate nomogram</article-title><source>Ann Surg</source><year>2007</year><volume>245</volume><fpage>462</fpage><lpage>468</lpage><pub-id pub-id-type="pmid">17435554</pub-id><pub-id pub-id-type="doi">10.1097/01.sla.0000250439.86020.85</pub-id></citation></ref><ref id="B17"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kocsis</surname><given-names>L</given-names></name><name><surname>Svebis</surname><given-names>M</given-names></name><name><surname>Boross</surname><given-names>G</given-names></name><name><surname>Sinko</surname><given-names>M</given-names></name><name><surname>Maraz</surname><given-names>R</given-names></name><name><surname>Rajtar</surname><given-names>M</given-names></name><name><surname>Cserni</surname><given-names>G</given-names></name></person-group><article-title>Use and limitations of a nomogram predicting the likelihood of non-sentinel node involvement after a positive sentinel node biopsy in breast cancer patients</article-title><source>Am J Surg</source><year>2004</year><volume>70</volume><fpage>1019</fpage><lpage>1024</lpage><pub-id pub-id-type="pmid">15586519</pub-id></citation></ref><ref id="B18"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Degnim</surname><given-names>AC</given-names></name><name><surname>Reynolds</surname><given-names>C</given-names></name><name><surname>Pantvaidya</surname><given-names>G</given-names></name><name><surname>Zakaria</surname><given-names>S</given-names></name><name><surname>Hoskin</surname><given-names>T</given-names></name><name><surname>Barnes</surname><given-names>S</given-names></name><name><surname>Roberts</surname><given-names>MV</given-names></name><name><surname>Lucas</surname><given-names>PC</given-names></name><name><surname>Oh</surname><given-names>K</given-names></name><name><surname>Koker</surname><given-names>M</given-names></name><name><surname>Sabel</surname><given-names>MS</given-names></name><name><surname>Newman</surname><given-names>LA</given-names></name></person-group><article-title>Nonsentinel node metastasis in breast cancer patients: assessment of an existing and a new predictive nomogram</article-title><source>Am J Surg</source><year>2005</year><volume>190</volume><fpage>543</fpage><lpage>550</lpage><pub-id pub-id-type="pmid">16164917</pub-id><pub-id pub-id-type="doi">10.1016/j.amjsurg.2005.06.008</pub-id></citation></ref><ref id="B19"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Alran</surname><given-names>S</given-names></name><name><surname>De Rycke</surname><given-names>Y</given-names></name><name><surname>Fourchotte</surname><given-names>V</given-names></name><name><surname>Charitansky</surname><given-names>H</given-names></name><name><surname>Laki</surname><given-names>F</given-names></name><name><surname>Falcou</surname><given-names>MC</given-names></name><name><surname>Benamor</surname><given-names>M</given-names></name><name><surname>Freneaux</surname><given-names>P</given-names></name><name><surname>Salmon</surname><given-names>RJ</given-names></name><name><surname>Sigal-Zifrani</surname><given-names>B</given-names></name></person-group><article-title>Validation and limitations of use of a breast cancer nomogram predicting the likelihood of non-sentinel node involvement after positive sentinel node biopsy</article-title><source>Ann Surg Oncol</source><year>2007</year><volume>14</volume><fpage>2195</fpage><lpage>2201</lpage><pub-id pub-id-type="pmid">17294071</pub-id><pub-id pub-id-type="doi">10.1245/s10434-006-9331-2</pub-id></citation></ref><ref id="B20"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cody</surname><given-names>HS</given-names><suffix>3rd</suffix></name><name><surname>Borgen</surname><given-names>PI</given-names></name></person-group><article-title>State-of-the-art approaches to sentinel node biopsy for breast cancer: study design, patient selection, technique, and quality control at Memorial Sloan-Kettering Cancer Center</article-title><source>Surg Oncol</source><year>1999</year><volume>8</volume><fpage>85</fpage><lpage>91</lpage><pub-id pub-id-type="pmid">10732960</pub-id><pub-id pub-id-type="doi">10.1016/S0960-7404(99)00029-8</pub-id></citation></ref><ref id="B21"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cserni</surname><given-names>G</given-names></name><name><surname>Amendoeira</surname><given-names>I</given-names></name><name><surname>Apostolikas</surname><given-names>N</given-names></name><name><surname>Bellocq</surname><given-names>JP</given-names></name><name><surname>Bianchi</surname><given-names>S</given-names></name><name><surname>Bussolati</surname><given-names>G</given-names></name><name><surname>Boecker</surname><given-names>W</given-names></name><name><surname>Borisch</surname><given-names>B</given-names></name><name><surname>Connolly</surname><given-names>CE</given-names></name><name><surname>Decker</surname><given-names>T</given-names></name><name><surname>Dervan</surname><given-names>P</given-names></name><name><surname>Drijkoningen</surname><given-names>M</given-names></name><name><surname>Ellis</surname><given-names>IO</given-names></name><name><surname>Elston</surname><given-names>CW</given-names></name><name><surname>Eusebi</surname><given-names>V</given-names></name><name><surname>Faverly</surname><given-names>D</given-names></name><name><surname>Heikkila</surname><given-names>P</given-names></name><name><surname>Holland</surname><given-names>R</given-names></name><name><surname>Kerner</surname><given-names>H</given-names></name><name><surname>Kulka</surname><given-names>J</given-names></name><name><surname>Jacquemier</surname><given-names>J</given-names></name><name><surname>Lacerda</surname><given-names>M</given-names></name><name><surname>Martinez-Penuela</surname><given-names>J</given-names></name><name><surname>De Miguel</surname><given-names>C</given-names></name><name><surname>Peterse</surname><given-names>JL</given-names></name><name><surname>Rank</surname><given-names>F</given-names></name><name><surname>Regitnig</surname><given-names>P</given-names></name><name><surname>Reiner</surname><given-names>A</given-names></name><name><surname>Sapino</surname><given-names>A</given-names></name><name><surname>Sigal-Zafrani</surname><given-names>B</given-names></name><name><surname>Tanous</surname><given-names>AM</given-names></name><name><surname>Thorstenson</surname><given-names>S</given-names></name><name><surname>Zozaya</surname><given-names>E</given-names></name><name><surname>Wells</surname><given-names>CA</given-names></name></person-group><article-title>Pathological work-up of sentinel lymph nodes in breast cancer. Review of current data to be considered for the formulation of guidelines</article-title><source>Eur J Cancer</source><year>2003</year><volume>39</volume><fpage>1654</fpage><lpage>1667</lpage><pub-id pub-id-type="pmid">12888359</pub-id><pub-id pub-id-type="doi">10.1016/S0959-8049(03)00203-X</pub-id></citation></ref><ref id="B22"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Elston</surname><given-names>CW</given-names></name><name><surname>Ellis</surname><given-names>IO</given-names></name></person-group><article-title>Pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up</article-title><source>Histopathology</source><year>1991</year><volume>19</volume><fpage>403</fpage><lpage>410</lpage><pub-id pub-id-type="pmid">1757079</pub-id><pub-id pub-id-type="doi">10.1111/j.1365-2559.1991.tb00229.x</pub-id></citation></ref><ref id="B23"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Singletary</surname><given-names>SE</given-names></name><name><surname>Greene</surname><given-names>FL</given-names></name><name><surname>Sobin</surname><given-names>LH</given-names></name></person-group><article-title>Classification of isolated tumor cells: clarification of the 6th edition of the American Joint Committee on Cancer Staging Manual</article-title><source>Cancer</source><year>2003</year><volume>98</volume><fpage>2740</fpage><lpage>2741</lpage><pub-id pub-id-type="pmid">14669301</pub-id><pub-id pub-id-type="doi">10.1002/cncr.11865</pub-id></citation></ref><ref id="B24"><citation citation-type="book"><person-group person-group-type="author"><name><surname>Breiman</surname><given-names>L</given-names></name><name><surname>Friedman</surname><given-names>JH</given-names></name><name><surname>Olshen</surname><given-names>RA</given-names></name><name><surname>Stone</surname><given-names>CJ</given-names></name></person-group><source>Classification and Regression Trees</source><year>1984</year><publisher-name>Belmont, CA , Wadsworth</publisher-name></citation></ref><ref id="B25"><citation citation-type="book"><person-group person-group-type="author"><name><surname>Dalgaard</surname><given-names>P</given-names></name></person-group><source>Introductory Statistics with R</source><year>2002</year><publisher-name>USA , Springer</publisher-name></citation></ref><ref id="B26"><citation citation-type="book"><person-group person-group-type="author"><name><surname>Kraemer</surname><given-names>HC</given-names></name></person-group><source>Evaluating medical tests: objective and quantitative guidelines</source><year>1992</year><publisher-name>Newbury Park, CA. , Sage Publications</publisher-name></citation></ref><ref id="B27"><citation citation-type="book"><person-group person-group-type="author"><name><surname>Hastie</surname><given-names>T</given-names></name><name><surname>Tibshirani</surname><given-names>R</given-names></name><name><surname>Friedman</surname><given-names>JH</given-names></name></person-group><source>The Elements of Statistical Learning</source><year>2001</year><publisher-name>USA , Springer</publisher-name></citation></ref><ref id="B28"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kwak</surname><given-names>LW</given-names></name><name><surname>Halpern</surname><given-names>J</given-names></name><name><surname>Olshen</surname><given-names>RA</given-names></name><name><surname>Horning</surname><given-names>SJ</given-names></name></person-group><article-title>Prognostic significance of actual dose intensity in diffuse large-cell lymphoma: results of a tree-structured survival analysis</article-title><source>J Clin Oncol</source><year>1990</year><volume>8</volume><fpage>963</fpage><lpage>977</lpage><pub-id pub-id-type="pmid">2348230</pub-id></citation></ref><ref id="B29"><citation citation-type="other"><article-title>ROC software, Sierra-Pacific MIRECC</article-title><ext-link ext-link-type="uri" xlink:href="http://mirecc.stanford.edu/"></ext-link></citation></ref><ref id="B30"><citation citation-type="other"><article-title>CART software, Salford Systems</article-title><ext-link ext-link-type="uri" xlink:href="http://www.salford-systems.com/cart.php"></ext-link></citation></ref><ref id="B31"><citation citation-type="book"><person-group person-group-type="author"><name><surname>McLachlan</surname><given-names>G</given-names></name></person-group><source>Discriminant Analysis and Statistical Pattern Recognition</source><year>1992</year><publisher-name>USA , John Wiley and Sons, Inc.</publisher-name></citation></ref><ref id="B32"><citation citation-type="book"><person-group person-group-type="author"><name><surname>Stone</surname><given-names>CJ</given-names></name></person-group><source>A Course in Probability and Statistics</source><year>1995</year><publisher-name>Belmont, CA , Duxbury Press</publisher-name></citation></ref><ref id="B33"><citation citation-type="book"><person-group person-group-type="author"><name><surname>Team</surname><given-names>RDC</given-names></name></person-group><source>R: A Language and Environment for Statistical Computing.</source><year>2006</year><publisher-name>Vienna, Austria , R Foundation for Statistical Computing</publisher-name><ext-link ext-link-type="uri" xlink:href="http://www.R-project.org"></ext-link><pub-id pub-id-type="pmid">16835509</pub-id></citation></ref><ref id="B34"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>DeLong</surname><given-names>ER</given-names></name><name><surname>DeLong</surname><given-names>DM</given-names></name><name><surname>Clarke-Pearson</surname><given-names>DL</given-names></name></person-group><article-title>Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach</article-title><source>Biometrics</source><year>1988</year><volume>44</volume><fpage>837</fpage><lpage>845</lpage><pub-id pub-id-type="pmid">3203132</pub-id><pub-id pub-id-type="doi">10.2307/2531595</pub-id></citation></ref><ref id="B35"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Veronesi</surname><given-names>U</given-names></name><name><surname>Paganelli</surname><given-names>G</given-names></name><name><surname>Viale</surname><given-names>G</given-names></name><name><surname>Luini</surname><given-names>A</given-names></name><name><surname>Zurrida</surname><given-names>S</given-names></name><name><surname>Galimberti</surname><given-names>V</given-names></name><name><surname>Intra</surname><given-names>M</given-names></name><name><surname>Veronesi</surname><given-names>P</given-names></name><name><surname>Robertson</surname><given-names>C</given-names></name><name><surname>Maisonneuve</surname><given-names>P</given-names></name><name><surname>Renne</surname><given-names>G</given-names></name><name><surname>De Cicco</surname><given-names>C</given-names></name><name><surname>De Lucia</surname><given-names>F</given-names></name><name><surname>Gennari</surname><given-names>R</given-names></name></person-group><article-title>A randomized comparison of sentinel-node biopsy with routine axillary dissection in breast cancer</article-title><source>N Engl J Med</source><year>2003</year><volume>349</volume><fpage>546</fpage><lpage>553</lpage><pub-id pub-id-type="pmid">12904519</pub-id><pub-id pub-id-type="doi">10.1056/NEJMoa012782</pub-id></citation></ref><ref id="B36"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Esserman</surname><given-names>L</given-names></name><name><surname>Sepucha</surname><given-names>K</given-names></name></person-group><article-title>Practice implications of the high false negative rate of sentinel lymph node biopsy reported in NSABP-32</article-title><source>J Clin Oncol</source><year>2005</year><volume>23</volume><fpage>812</fpage></citation></ref><ref id="B37"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Veronesi</surname><given-names>U</given-names></name><name><surname>Paganelli</surname><given-names>G</given-names></name><name><surname>Viale</surname><given-names>G</given-names></name><name><surname>Luini</surname><given-names>A</given-names></name><name><surname>Zurrida</surname><given-names>S</given-names></name><name><surname>Galimberti</surname><given-names>V</given-names></name><name><surname>Intra</surname><given-names>M</given-names></name><name><surname>Veronesi</surname><given-names>P</given-names></name><name><surname>Maisonneuve</surname><given-names>P</given-names></name><name><surname>Gatti</surname><given-names>G</given-names></name><name><surname>Mazzarol</surname><given-names>G</given-names></name><name><surname>De Cicco</surname><given-names>C</given-names></name><name><surname>Manfredi</surname><given-names>G</given-names></name><name><surname>Fernandez</surname><given-names>JR</given-names></name></person-group><article-title>Sentinel-lymph-node biopsy as a staging procedure in breast cancer: update of a randomised controlled study</article-title><source>Lancet Onc</source><year>2006</year><volume>7</volume><fpage>983</fpage><lpage>990</lpage><pub-id pub-id-type="doi">10.1016/S1470-2045(06)70947-0</pub-id></citation></ref><ref id="B38"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>van Rijk</surname><given-names>MC</given-names></name><name><surname>Peterse</surname><given-names>JL</given-names></name><name><surname>Nieweg</surname><given-names>OE</given-names></name><name><surname>Oldenburg</surname><given-names>HS</given-names></name><name><surname>Rutgers</surname><given-names>EJ</given-names></name><name><surname>Kroon</surname><given-names>BB</given-names></name></person-group><article-title>Additional axillary metastases and stage migration in breast cancer patients with micrometastases or submicrometastases in sentinel lymph nodes</article-title><source>Cancer</source><year>2006</year><volume>107</volume><fpage>467</fpage><lpage>471</lpage><pub-id pub-id-type="pmid">16804924</pub-id><pub-id pub-id-type="doi">10.1002/cncr.22069</pub-id></citation></ref><ref id="B39"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Orr</surname><given-names>RK</given-names></name></person-group><article-title>The impact of prophylactic axillary node dissection on breast cancer survival--a Bayesian meta-analysis</article-title><source>Ann Surg Oncol</source><year>1999</year><volume>6</volume><fpage>109</fpage><lpage>116</lpage><pub-id pub-id-type="pmid">10030423</pub-id><pub-id pub-id-type="doi">10.1007/s10434-999-0109-1</pub-id></citation></ref><ref id="B40"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Krag</surname><given-names>DN</given-names></name><name><surname>Julian</surname><given-names>TB</given-names></name><name><surname>Harlow</surname><given-names>SP</given-names></name><name><surname>Weaver</surname><given-names>DL</given-names></name><name><surname>Ashikaga</surname><given-names>T</given-names></name><name><surname>Bryant</surname><given-names>J</given-names></name><name><surname>Single</surname><given-names>RM</given-names></name><name><surname>Wolmark</surname><given-names>N</given-names></name></person-group><article-title>NSABP-32: Phase III, randomized trial comparing axillary resection with sentinal lymph node dissection: a description of the trial</article-title><source>Ann Surg Oncol</source><year>2004</year><volume>11</volume><fpage>208S</fpage><lpage>10S</lpage><pub-id pub-id-type="pmid">15023753</pub-id></citation></ref><ref id="B41"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Grube</surname><given-names>BJ</given-names></name><name><surname>Giuliano</surname><given-names>AE</given-names></name></person-group><article-title>Observation of the breast cancer patient with a tumor-positive sentinel node: implications of the ACOSOG Z0011 trial</article-title><source>Semin Surg Oncol</source><year>2001</year><volume>20</volume><fpage>230</fpage><lpage>237</lpage><pub-id pub-id-type="pmid">11523108</pub-id><pub-id pub-id-type="doi">10.1002/ssu.1038</pub-id></citation></ref><ref id="B42"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Fant</surname><given-names>JS</given-names></name><name><surname>Grant</surname><given-names>MD</given-names></name><name><surname>Knox</surname><given-names>SM</given-names></name><name><surname>Livingston</surname><given-names>SA</given-names></name><name><surname>Ridl</surname><given-names>K</given-names></name><name><surname>Jones</surname><given-names>RC</given-names></name><name><surname>Kuhn</surname><given-names>JA</given-names></name></person-group><article-title>Preliminary outcome analysis in patients with breast cancer and a positive sentinel lymph node who declined axillary dissection</article-title><source>Ann Surg Oncol</source><year>2003</year><volume>10</volume><fpage>126</fpage><lpage>130</lpage><pub-id pub-id-type="pmid">12620906</pub-id><pub-id pub-id-type="doi">10.1245/ASO.2003.04.022</pub-id></citation></ref><ref id="B43"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wada</surname><given-names>N</given-names></name><name><surname>Imoto</surname><given-names>S</given-names></name><name><surname>Yamauchi</surname><given-names>C</given-names></name><name><surname>Hasebe</surname><given-names>T</given-names></name><name><surname>Ochiai</surname><given-names>A</given-names></name></person-group><article-title>Predictors of tumour involvement in remaining axillary lymph nodes of breast cancer patients with positive sentinel lymph node</article-title><source>Eur J Surg Oncol</source><year>2006</year><volume>32</volume><fpage>29</fpage><lpage>33</lpage><pub-id pub-id-type="pmid">16269227</pub-id><pub-id pub-id-type="doi">10.1016/j.ejso.2005.08.010</pub-id></citation></ref><ref id="B44"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kamath</surname><given-names>VJ</given-names></name><name><surname>Giuliano</surname><given-names>R</given-names></name><name><surname>Dauway</surname><given-names>EL</given-names></name><name><surname>Cantor</surname><given-names>A</given-names></name><name><surname>Berman</surname><given-names>C</given-names></name><name><surname>Ku</surname><given-names>NN</given-names></name><name><surname>Cox</surname><given-names>CE</given-names></name><name><surname>Reintgen</surname><given-names>DS</given-names></name></person-group><article-title>Characteristics of the sentinel lymph node in breast cancer predict further involvement of higher-echelon nodes in the axilla: a study to evaluate the need for complete axillary lymph node dissection</article-title><source>Arch Surg</source><year>2001</year><volume>136</volume><fpage>688</fpage><lpage>692</lpage><pub-id pub-id-type="pmid">11387010</pub-id><pub-id pub-id-type="doi">10.1001/archsurg.136.6.688</pub-id></citation></ref><ref id="B45"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Yu</surname><given-names>JC</given-names></name><name><surname>Hsu</surname><given-names>GC</given-names></name><name><surname>Hsieh</surname><given-names>CB</given-names></name><name><surname>Sheu</surname><given-names>LF</given-names></name><name><surname>Chao</surname><given-names>TY</given-names></name></person-group><article-title>Prediction of metastasis to non-sentinel nodes by sentinel node status and primary tumor characteristics in primary breast cancer in Taiwan</article-title><source>World J Surg</source><year>2005</year><volume>29</volume><fpage>813</fpage><lpage>8; discussion 818-9</lpage><pub-id pub-id-type="pmid">15951935</pub-id><pub-id pub-id-type="doi">10.1007/s00268-005-7744-x</pub-id></citation></ref><ref id="B46"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Viale</surname><given-names>G</given-names></name><name><surname>Maiorano</surname><given-names>E</given-names></name><name><surname>Pruneri</surname><given-names>G</given-names></name><name><surname>Mastropasqua</surname><given-names>MG</given-names></name><name><surname>Valentini</surname><given-names>S</given-names></name><name><surname>Galimberti</surname><given-names>V</given-names></name><name><surname>Zurrida</surname><given-names>S</given-names></name><name><surname>Maisonneuve</surname><given-names>P</given-names></name><name><surname>Paganelli</surname><given-names>G</given-names></name><name><surname>Mazzarol</surname><given-names>G</given-names></name></person-group><article-title>Predicting the risk for additional axillary metastases in patients with breast carcinoma and positive sentinel lymph node biopsy</article-title><source>Ann Surg</source><year>2005</year><volume>241</volume><fpage>319</fpage><lpage>325</lpage><pub-id pub-id-type="pmid">15650643</pub-id><pub-id pub-id-type="doi">10.1097/01.sla.0000150255.30665.52</pub-id></citation></ref><ref id="B47"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bolster</surname><given-names>MJ</given-names></name><name><surname>Peer</surname><given-names>PGM</given-names></name><name><surname>Bult</surname><given-names>P</given-names></name><name><surname>Thunnissen</surname><given-names>FBJM</given-names></name><name><surname>Schapers</surname><given-names>RFM</given-names></name><name><surname>Meijer</surname><given-names>JWR</given-names></name><name><surname>Strobbe</surname><given-names>LJA</given-names></name><name><surname>van Berlo</surname><given-names>CLH</given-names></name><name><surname>Klinkenbijl</surname><given-names>JHG</given-names></name><name><surname>Beex</surname><given-names>LVAM</given-names></name><name><surname>Wobbes</surname><given-names>T</given-names></name><name><surname>Tjan-Heijnen</surname><given-names>VCG</given-names></name></person-group><article-title>Risk factors for non-sentinel lymph node metastases in patients with breast cancer. The outcome of a multi-institutional study</article-title><source>Ann Surg Oncol</source><year>2007</year><volume>14</volume><fpage>181</fpage><lpage>189</lpage><pub-id pub-id-type="pmid">17028772</pub-id><pub-id pub-id-type="doi">10.1245/s10434-006-9065-1</pub-id></citation></ref><ref id="B48"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Carcoforo</surname><given-names>P</given-names></name><name><surname>Maestroni</surname><given-names>U</given-names></name><name><surname>Querzoli</surname><given-names>P</given-names></name><name><surname>Lanzara</surname><given-names>S</given-names></name><name><surname>Maravegias</surname><given-names>K</given-names></name><name><surname>Feggi</surname><given-names>L</given-names></name><name><surname>Soliani</surname><given-names>G</given-names></name><name><surname>Basaglia</surname><given-names>E</given-names></name></person-group><article-title>Primary breast cancer features can predict additional lymph node involvement in patients with sentinel node micrometastases</article-title><source>World J Surg</source><year>2006</year><volume>30</volume><fpage>1653</fpage><lpage>1657</lpage><pub-id pub-id-type="pmid">16927059</pub-id><pub-id pub-id-type="doi">10.1007/s00268-005-0083-0</pub-id></citation></ref><ref id="B49"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rivers</surname><given-names>AK</given-names></name><name><surname>Griffith</surname><given-names>KA</given-names></name><name><surname>Hunt</surname><given-names>KK</given-names></name><name><surname>Degnim</surname><given-names>AC</given-names></name><name><surname>Sabel</surname><given-names>MS</given-names></name><name><surname>Diehl</surname><given-names>KM</given-names></name><name><surname>Cimmino</surname><given-names>VM</given-names></name><name><surname>Chang</surname><given-names>AE</given-names></name><name><surname>Lucas</surname><given-names>PC</given-names></name><name><surname>Newman</surname><given-names>LA</given-names></name></person-group><article-title>Clinicopathologic features associated with having four or more metastatic axillary nodes in breast cancer patients with a positive sentinel lymph node</article-title><source>Ann Surg Oncol</source><year>2006</year><volume>13</volume><fpage>36</fpage><lpage>44</lpage><pub-id pub-id-type="pmid">16378156</pub-id><pub-id pub-id-type="doi">10.1245/ASO.2006.03.080</pub-id></citation></ref><ref id="B50"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ozmen</surname><given-names>V</given-names></name><name><surname>Karanlik</surname><given-names>H</given-names></name><name><surname>Cabioglu</surname><given-names>N</given-names></name><name><surname>Igci</surname><given-names>A</given-names></name><name><surname>Kecer</surname><given-names>M</given-names></name><name><surname>Asoglu</surname><given-names>O</given-names></name><name><surname>Tuzlali</surname><given-names>S</given-names></name><name><surname>Mudun</surname><given-names>A</given-names></name></person-group><article-title>Factors predicting the sentinel and non-sentinel lymph node metastases in breast cancer</article-title><source>Breast Cancer Res Treat</source><year>2006</year><volume>95</volume><fpage>1</fpage><lpage>6</lpage><pub-id pub-id-type="pmid">16322900</pub-id><pub-id pub-id-type="doi">10.1007/s10549-005-9007-9</pub-id></citation></ref><ref id="B51"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Calhoun</surname><given-names>KE</given-names></name><name><surname>Hansen</surname><given-names>NM</given-names></name><name><surname>Turner</surname><given-names>RR</given-names></name><name><surname>Giuliano</surname><given-names>AE</given-names></name></person-group><article-title>Nonsentinel node metastases in breast cancer patients with isolated tumor cells in the sentinel node: implications for completion axillary node dissection</article-title><source>Am J Surg</source><year>2005</year><volume>190</volume><fpage>588</fpage><lpage>591</lpage><pub-id pub-id-type="pmid">16164927</pub-id><pub-id pub-id-type="doi">10.1016/j.amjsurg.2005.06.018</pub-id></citation></ref><ref id="B52"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cserni</surname><given-names>G</given-names></name><name><surname>Gregori</surname><given-names>D</given-names></name><name><surname>Merletti</surname><given-names>F</given-names></name><name><surname>Sapino</surname><given-names>A</given-names></name><name><surname>Mano</surname><given-names>MP</given-names></name><name><surname>Ponti</surname><given-names>A</given-names></name><name><surname>Sandrucci</surname><given-names>S</given-names></name><name><surname>Baltas</surname><given-names>B</given-names></name><name><surname>Bussolati</surname><given-names>G</given-names></name></person-group><article-title>Meta-analysis of non-sentinel node metastases associated with micrometastatic sentinel nodes in breast cancer</article-title><source>Br J Surg</source><year>2004</year><volume>91</volume><fpage>1245</fpage><lpage>1252</lpage><pub-id pub-id-type="pmid">15376203</pub-id><pub-id pub-id-type="doi">10.1002/bjs.4725</pub-id></citation></ref><ref id="B53"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cserni</surname><given-names>G</given-names></name><name><surname>Bianchi</surname><given-names>S</given-names></name><name><surname>Vezzosi</surname><given-names>V</given-names></name><name><surname>Arisio</surname><given-names>R</given-names></name><name><surname>Bori</surname><given-names>R</given-names></name><name><surname>Peterse</surname><given-names>JL</given-names></name><name><surname>Sapino</surname><given-names>A</given-names></name><name><surname>Castellano</surname><given-names>I</given-names></name><name><surname>Drijkoningen</surname><given-names>M</given-names></name><name><surname>Kulka</surname><given-names>J</given-names></name><name><surname>Eusebi</surname><given-names>V</given-names></name><name><surname>Foschini</surname><given-names>MP</given-names></name><name><surname>Bellocq</surname><given-names>JP</given-names></name><name><surname>Marin</surname><given-names>C</given-names></name><name><surname>Thorstenson</surname><given-names>S</given-names></name><name><surname>Amendoeira</surname><given-names>I</given-names></name><name><surname>Reiner-Concin</surname><given-names>A</given-names></name><name><surname>Decker</surname><given-names>T</given-names></name><name><surname>Lacerda</surname><given-names>M</given-names></name><name><surname>Figueiredo</surname><given-names>P</given-names></name><name><surname>Fejes</surname><given-names>G</given-names></name></person-group><article-title>Sentinel lymph node biopsy in staging small (up to 15 mm) breast carcinomas. Results from a European multi-institutional study</article-title><source>Path Onc Res</source><year>2007</year><volume>13</volume><fpage>5</fpage><lpage>14</lpage></citation></ref><ref id="B54"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kohrt</surname><given-names>HE</given-names></name><name><surname>Nouri</surname><given-names>N</given-names></name><name><surname>Nowels</surname><given-names>K</given-names></name><name><surname>Johnson</surname><given-names>D</given-names></name><name><surname>Holmes</surname><given-names>S</given-names></name><name><surname>Lee</surname><given-names>PP</given-names></name></person-group><article-title>Profile of immune cells in axillary lymph nodes predicts disease-free survival in breast cancer</article-title><source>PLoS Med</source><year>2005</year><volume>2</volume><fpage>e284</fpage><pub-id pub-id-type="pmid">16124834</pub-id><pub-id pub-id-type="doi">10.1371/journal.pmed.0020284</pub-id></citation></ref><ref id="B55"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Jeffrey</surname><given-names>SS</given-names></name><name><surname>Lonning</surname><given-names>PE</given-names></name><name><surname>Hillner</surname><given-names>BE</given-names></name></person-group><article-title>Genomics-based prognosis and therapeutic prediction in breast cancer</article-title><source>J Natl Compr Canc Netw</source><year>2005</year><volume>3</volume><fpage>291</fpage><lpage>300</lpage><pub-id pub-id-type="pmid">16002001</pub-id></citation></ref><ref id="B56"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Soni</surname><given-names>NK</given-names></name><name><surname>Carmalt</surname><given-names>HL</given-names></name><name><surname>Gillett</surname><given-names>DJ</given-names></name><name><surname>Spillane</surname><given-names>AJ</given-names></name></person-group><article-title>Evaluation of a breast cancer nomogram for prediction of non-sentinel lymph node positivity</article-title><source>Eur J Surg Oncol</source><year>2005</year><volume>31</volume><fpage>958</fpage><lpage>964</lpage><pub-id pub-id-type="pmid">15979270</pub-id><pub-id pub-id-type="doi">10.1016/j.ejso.2005.04.011</pub-id></citation></ref><ref id="B57"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lambert</surname><given-names>LA</given-names></name><name><surname>Ayers</surname><given-names>GD</given-names></name><name><surname>Hwang</surname><given-names>RF</given-names></name><name><surname>Hunt</surname><given-names>KK</given-names></name><name><surname>Ross</surname><given-names>MI</given-names></name><name><surname>Kuerer</surname><given-names>HM</given-names></name><name><surname>Singletary</surname><given-names>SE</given-names></name><name><surname>Babiera</surname><given-names>GV</given-names></name><name><surname>Ames</surname><given-names>FC</given-names></name><name><surname>Feig</surname><given-names>B</given-names></name><name><surname>Lucci</surname><given-names>A</given-names></name><name><surname>Krishnamurthy</surname><given-names>S</given-names></name><name><surname>Meric-Bernstam</surname><given-names>F</given-names></name></person-group><article-title>Validation of a breast cancer nomogram for predicting nonsentinel lymph node metastases after a positive sentinel node biopsy</article-title><source>Ann Surg Oncol</source><year>2006</year><volume>13</volume><fpage>310</fpage><lpage>320</lpage><pub-id pub-id-type="pmid">16463073</pub-id><pub-id pub-id-type="doi">10.1245/ASO.2006.03.078</pub-id></citation></ref><ref id="B58"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cripe</surname><given-names>MH</given-names></name><name><surname>Beran</surname><given-names>LC</given-names></name><name><surname>Liang</surname><given-names>WC</given-names></name><name><surname>Sickle-Santanello</surname><given-names>BJ</given-names></name></person-group><article-title>The likelihood of additional nodal disease following a positive sentinel lymph node biopsy in breast cancer patients: validation of a nomogram</article-title><source>Am J Surg</source><year>2006</year><volume>192</volume><fpage>484</fpage><pub-id pub-id-type="pmid">16978955</pub-id><pub-id pub-id-type="doi">10.1016/j.amjsurg.2006.06.016</pub-id></citation></ref><ref id="B59"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Dauphine</surname><given-names>CE</given-names></name><name><surname>Haukoos</surname><given-names>JS</given-names></name><name><surname>Vargas</surname><given-names>MP</given-names></name><name><surname>Isaac</surname><given-names>NM</given-names></name><name><surname>Khalkhali</surname><given-names>I</given-names></name><name><surname>Vargas</surname><given-names>HI</given-names></name></person-group><article-title>Evaluation of three scoring systems predicting non sentinel node metastasis in breast cancer patients with a positive sentinel node biopsy</article-title><source>Ann Surg Oncol</source><year>2007</year><volume>14</volume><fpage>1014</fpage><lpage>1019</lpage><pub-id pub-id-type="pmid">17176986</pub-id><pub-id pub-id-type="doi">10.1245/s10434-006-9223-5</pub-id></citation></ref><ref id="B60"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ponzone</surname><given-names>R</given-names></name><name><surname>Maggiorotto</surname><given-names>F</given-names></name><name><surname>Mariani</surname><given-names>L</given-names></name><name><surname>Jacomuzzi</surname><given-names>ME</given-names></name><name><surname>Magistris</surname><given-names>A</given-names></name><name><surname>Mininanni</surname><given-names>P</given-names></name><name><surname>Biglia</surname><given-names>N</given-names></name><name><surname>Sismondi</surname><given-names>P</given-names></name></person-group><article-title>Comparison of two models for the prediction of nonsentinel node metastases in breast cancer</article-title><source>Am J Surg</source><year>2007</year><volume>193</volume><fpage>686</fpage><lpage>692</lpage><pub-id pub-id-type="pmid">17512277</pub-id><pub-id pub-id-type="doi">10.1016/j.amjsurg.2006.09.031</pub-id></citation></ref><ref id="B61"><citation citation-type="other"><article-title>Surveillance Epidemiology and End Results Statistics</article-title><ext-link ext-link-type="uri" xlink:href="http://seer.cancer.gov/"></ext-link></citation></ref><ref id="B62"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hwang</surname><given-names>RF</given-names></name><name><surname>Krishnamurthy</surname><given-names>S</given-names></name><name><surname>Hunt</surname><given-names>KK</given-names></name><name><surname>Mirza</surname><given-names>N</given-names></name><name><surname>Ames</surname><given-names>FC</given-names></name><name><surname>Feig</surname><given-names>B</given-names></name><name><surname>Kuerer</surname><given-names>HM</given-names></name><name><surname>Singletary</surname><given-names>SE</given-names></name><name><surname>Babiera</surname><given-names>G</given-names></name><name><surname>Meric</surname><given-names>F</given-names></name><name><surname>Akins</surname><given-names>JS</given-names></name><name><surname>Neely</surname><given-names>J</given-names></name><name><surname>Ross</surname><given-names>MI</given-names></name></person-group><article-title>Clinicopathologic factors predicting involvement of nonsentinel axillary nodes in women with breast cancer</article-title><source>Ann Surg Oncol</source><year>2003</year><volume>10</volume><fpage>248</fpage><lpage>254</lpage><pub-id pub-id-type="pmid">12679309</pub-id><pub-id pub-id-type="doi">10.1245/ASO.2003.05.020</pub-id></citation></ref><ref id="B63"><citation citation-type="journal"><person-group person-group-type="author"><name><surname>Barranger</surname><given-names>E</given-names></name><name><surname>Coutant</surname><given-names>C</given-names></name><name><surname>Flahault</surname><given-names>A</given-names></name><name><surname>Delpech</surname><given-names>Y</given-names></name><name><surname>Darai</surname><given-names>E</given-names></name><name><surname>Uzan</surname><given-names>S</given-names></name></person-group><article-title>An axilla scoring system to predict non-sentinel lymph node status in breast cancer patients with sentinel lymph node involvement</article-title><source>Breast Cancer Res Treat</source><year>2005</year><volume>91</volume><fpage>113</fpage><lpage>119</lpage><pub-id pub-id-type="pmid">15868438</pub-id><pub-id pub-id-type="doi">10.1007/s10549-004-5781-z</pub-id></citation></ref><ref id="B64"><citation citation-type="other"><article-title>Stanford Online Calculator</article-title><ext-link ext-link-type="uri" xlink:href="http://www-stat.stanford.edu/~olshen/NSLNcalculator/"></ext-link></citation></ref></ref-list></back></pmc></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Sante/explor/LymphedemaV1/Data/Pmc/Corpus
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 001B05 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd -nk 001B05 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Sante
|area= LymphedemaV1
|flux= Pmc
|étape= Corpus
|type= RBID
|clé= PMC:2311316
|texte= New models and online calculator for predicting non-sentinel lymph node status in sentinel lymph node positive breast cancer patients
}}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/RBID.i -Sk "pubmed:18315887" \
| HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd \
| NlmPubMed2Wicri -a LymphedemaV1
| This area was generated with Dilib version V0.6.31. |  |