A novel measure and significance testing in data analysis of cell image segmentation
Identifieur interne : 000265 ( Pmc/Corpus ); précédent : 000264; suivant : 000266A novel measure and significance testing in data analysis of cell image segmentation
Auteurs : Jin Chu Wu ; Michael Halter ; Raghu N. Kacker ; John T. Elliott ; Anne L. PlantSource :
- BMC Bioinformatics [ 1471-2105 ] ; 2017.
Abstract
Cell image segmentation (CIS) is an essential part of quantitative imaging of biological cells. Designing a performance measure and conducting significance testing are critical for evaluating and comparing the CIS algorithms for image-based cell assays in cytometry. Many measures and methods have been proposed and implemented to evaluate segmentation methods. However, computing the standard errors (SE) of the measures and their correlation coefficient is not described, and thus the statistical significance of performance differences between CIS algorithms cannot be assessed.
We propose the total error rate (TER), a novel performance measure for segmenting all cells in the supervised evaluation. The TER statistically aggregates all misclassification error rates (MER) by taking cell sizes as weights. The MERs are for segmenting each single cell in the population. The TER is fully supported by the pairwise comparisons of MERs using 106 manually segmented ground-truth cells with different sizes and seven CIS algorithms taken from ImageJ. Further, the SE and 95% confidence interval (CI) of TER are computed based on the SE of MER that is calculated using the bootstrap method. An algorithm for computing the correlation coefficient of TERs between two CIS algorithms is also provided. Hence, the 95% CI error bars can be used to classify CIS algorithms. The SEs of TERs and their correlation coefficient can be employed to conduct the hypothesis testing, while the CIs overlap, to determine the statistical significance of the performance differences between CIS algorithms.
A novel measure TER of CIS is proposed. The TER’s SEs and correlation coefficient are computed. Thereafter, CIS algorithms can be evaluated and compared statistically by conducting the significance testing.
Url:
DOI: 10.1186/s12859-017-1527-x
PubMed: 28292256
PubMed Central: 5351215
Links to Exploration step
PMC:5351215Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">A novel measure and significance testing in data analysis of cell image segmentation</title>
<author><name sortKey="Wu, Jin Chu" sort="Wu, Jin Chu" uniqKey="Wu J" first="Jin Chu" last="Wu">Jin Chu Wu</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
<author><name sortKey="Halter, Michael" sort="Halter, Michael" uniqKey="Halter M" first="Michael" last="Halter">Michael Halter</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
<author><name sortKey="Kacker, Raghu N" sort="Kacker, Raghu N" uniqKey="Kacker R" first="Raghu N." last="Kacker">Raghu N. Kacker</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
<author><name sortKey="Elliott, John T" sort="Elliott, John T" uniqKey="Elliott J" first="John T." last="Elliott">John T. Elliott</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
<author><name sortKey="Plant, Anne L" sort="Plant, Anne L" uniqKey="Plant A" first="Anne L." last="Plant">Anne L. Plant</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PMC</idno>
<idno type="pmid">28292256</idno>
<idno type="pmc">5351215</idno>
<idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5351215</idno>
<idno type="RBID">PMC:5351215</idno>
<idno type="doi">10.1186/s12859-017-1527-x</idno>
<date when="2017">2017</date>
<idno type="wicri:Area/Pmc/Corpus">000265</idno>
<idno type="wicri:explorRef" wicri:stream="Pmc" wicri:step="Corpus" wicri:corpus="PMC">000265</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">A novel measure and significance testing in data analysis of cell image segmentation</title>
<author><name sortKey="Wu, Jin Chu" sort="Wu, Jin Chu" uniqKey="Wu J" first="Jin Chu" last="Wu">Jin Chu Wu</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
<author><name sortKey="Halter, Michael" sort="Halter, Michael" uniqKey="Halter M" first="Michael" last="Halter">Michael Halter</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
<author><name sortKey="Kacker, Raghu N" sort="Kacker, Raghu N" uniqKey="Kacker R" first="Raghu N." last="Kacker">Raghu N. Kacker</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
<author><name sortKey="Elliott, John T" sort="Elliott, John T" uniqKey="Elliott J" first="John T." last="Elliott">John T. Elliott</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
<author><name sortKey="Plant, Anne L" sort="Plant, Anne L" uniqKey="Plant A" first="Anne L." last="Plant">Anne L. Plant</name>
<affiliation><nlm:aff id="Aff1"></nlm:aff>
</affiliation>
</author>
</analytic>
<series><title level="j">BMC Bioinformatics</title>
<idno type="eISSN">1471-2105</idno>
<imprint><date when="2017">2017</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en"><sec><title>Background</title>
<p>Cell image segmentation (CIS) is an essential part of quantitative imaging of biological cells. Designing a performance measure and conducting significance testing are critical for evaluating and comparing the CIS algorithms for image-based cell assays in cytometry. Many measures and methods have been proposed and implemented to evaluate segmentation methods. However, computing the standard errors (SE) of the measures and their correlation coefficient is not described, and thus the statistical significance of performance differences between CIS algorithms cannot be assessed.</p>
</sec>
<sec><title>Results</title>
<p>We propose the total error rate (TER), a novel performance measure for segmenting all cells in the supervised evaluation. The TER statistically aggregates all misclassification error rates (MER) by taking cell sizes as weights. The MERs are for segmenting each single cell in the population. The TER is fully supported by the pairwise comparisons of MERs using 106 manually segmented ground-truth cells with different sizes and seven CIS algorithms taken from ImageJ. Further, the SE and 95% confidence interval (CI) of TER are computed based on the SE of MER that is calculated using the bootstrap method. An algorithm for computing the correlation coefficient of TERs between two CIS algorithms is also provided. Hence, the 95% CI error bars can be used to classify CIS algorithms. The SEs of TERs and their correlation coefficient can be employed to conduct the hypothesis testing, while the CIs overlap, to determine the statistical significance of the performance differences between CIS algorithms.</p>
</sec>
<sec><title>Conclusions</title>
<p>A novel measure TER of CIS is proposed. The TER’s SEs and correlation coefficient are computed. Thereafter, CIS algorithms can be evaluated and compared statistically by conducting the significance testing.</p>
</sec>
</div>
</front>
<back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Hill, Aa" uniqKey="Hill A">AA Hill</name>
</author>
<author><name sortKey="Lapan, P" uniqKey="Lapan P">P LaPan</name>
</author>
<author><name sortKey="Li, Y" uniqKey="Li Y">Y Li</name>
</author>
<author><name sortKey="Haney, S" uniqKey="Haney S">S Haney</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Chalfoun, J" uniqKey="Chalfoun J">J Chalfoun</name>
</author>
<author><name sortKey="Kociolek, M" uniqKey="Kociolek M">M Kociolek</name>
</author>
<author><name sortKey="Dima, A" uniqKey="Dima A">A Dima</name>
</author>
<author><name sortKey="Cardone, A" uniqKey="Cardone A">A Cardone</name>
</author>
<author><name sortKey="Peskin, A" uniqKey="Peskin A">A Peskin</name>
</author>
<author><name sortKey="Bajcsy, P" uniqKey="Bajcsy P">P Bajcsy</name>
</author>
<author><name sortKey="Brady, M" uniqKey="Brady M">M Brady</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Bajcsy, P" uniqKey="Bajcsy P">P Bajcsy</name>
</author>
<author><name sortKey="Cardone, A" uniqKey="Cardone A">A Cardone</name>
</author>
<author><name sortKey="Chalfoun, J" uniqKey="Chalfoun J">J Chalfoun</name>
</author>
<author><name sortKey="Halter, M" uniqKey="Halter M">M Halter</name>
</author>
<author><name sortKey="Juba, D" uniqKey="Juba D">D Juba</name>
</author>
<author><name sortKey="Kociolek, M" uniqKey="Kociolek M">M Kociolek</name>
</author>
<author><name sortKey="Majurski, M" uniqKey="Majurski M">M Majurski</name>
</author>
<author><name sortKey="Peskin, A" uniqKey="Peskin A">A Peskin</name>
</author>
<author><name sortKey="Simon, Cg" uniqKey="Simon C">CG Simon</name>
</author>
<author><name sortKey="Simon, M" uniqKey="Simon M">M Simon</name>
</author>
<author><name sortKey="Vandecreme, A" uniqKey="Vandecreme A">A Vandecreme</name>
</author>
<author><name sortKey="Brady, M" uniqKey="Brady M">M Brady</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Ostle, B" uniqKey="Ostle B">B Ostle</name>
</author>
<author><name sortKey="Malone, Lc" uniqKey="Malone L">LC Malone</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Rand, Wm" uniqKey="Rand W">WM Rand</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Unnikrishnan, R" uniqKey="Unnikrishnan R">R Unnikrishnan</name>
</author>
<author><name sortKey="Pantofaru, C" uniqKey="Pantofaru C">C Pantofaru</name>
</author>
<author><name sortKey="Hebert, M" uniqKey="Hebert M">M Hebert</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Zijdenbos, Ap" uniqKey="Zijdenbos A">AP Zijdenbos</name>
</author>
<author><name sortKey="Dawant, Bm" uniqKey="Dawant B">BM Dawant</name>
</author>
<author><name sortKey="Margolin, Ra" uniqKey="Margolin R">RA Margolin</name>
</author>
<author><name sortKey="Palmer, Ac" uniqKey="Palmer A">AC Palmer</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Zhang, Yj" uniqKey="Zhang Y">YJ Zhang</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Klar, N" uniqKey="Klar N">N Klar</name>
</author>
<author><name sortKey="Lipsitz, Sr" uniqKey="Lipsitz S">SR Lipsitz</name>
</author>
<author><name sortKey="Parzen, M" uniqKey="Parzen M">M Parzen</name>
</author>
<author><name sortKey="Leong, T" uniqKey="Leong T">T Leong</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Dimopoulos, S" uniqKey="Dimopoulos S">S Dimopoulos</name>
</author>
<author><name sortKey="Mayer, Ce" uniqKey="Mayer C">CE Mayer</name>
</author>
<author><name sortKey="Rudolf, F" uniqKey="Rudolf F">F Rudolf</name>
</author>
<author><name sortKey="Stelling, J" uniqKey="Stelling J">J Stelling</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Zhang, H" uniqKey="Zhang H">H Zhang</name>
</author>
<author><name sortKey="Fritts, Je" uniqKey="Fritts J">JE Fritts</name>
</author>
<author><name sortKey="Goldman, Sa" uniqKey="Goldman S">SA Goldman</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Efron, B" uniqKey="Efron B">B Efron</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Efron, B" uniqKey="Efron B">B Efron</name>
</author>
<author><name sortKey="Tibshirani, Rj" uniqKey="Tibshirani R">RJ Tibshirani</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wu, Jc" uniqKey="Wu J">JC Wu</name>
</author>
<author><name sortKey="Martin, Af" uniqKey="Martin A">AF Martin</name>
</author>
<author><name sortKey="Kacker, Rn" uniqKey="Kacker R">RN Kacker</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wu, Jc" uniqKey="Wu J">JC Wu</name>
</author>
<author><name sortKey="Martin, Af" uniqKey="Martin A">AF Martin</name>
</author>
<author><name sortKey="Kacker, Rn" uniqKey="Kacker R">RN Kacker</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wu, Jc" uniqKey="Wu J">JC Wu</name>
</author>
<author><name sortKey="Martin, Af" uniqKey="Martin A">AF Martin</name>
</author>
<author><name sortKey="Kacker, Rn" uniqKey="Kacker R">RN Kacker</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Dresen, Img" uniqKey="Dresen I">IMG Dresen</name>
</author>
<author><name sortKey="Boes, T" uniqKey="Boes T">T Boes</name>
</author>
<author><name sortKey="Huesing, J" uniqKey="Huesing J">J Huesing</name>
</author>
<author><name sortKey="Neuhaeuser, M" uniqKey="Neuhaeuser M">M Neuhaeuser</name>
</author>
<author><name sortKey="Joeckel, Kh" uniqKey="Joeckel K">KH Joeckel</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Hand, Dj" uniqKey="Hand D">DJ Hand</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Van Der Waerden, Bl" uniqKey="Van Der Waerden B">BL van der Waerden</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Rice, Ja" uniqKey="Rice J">JA Rice</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Hyndman, Rj" uniqKey="Hyndman R">RJ Hyndman</name>
</author>
<author><name sortKey="Fan, Y" uniqKey="Fan Y">Y Fan</name>
</author>
</analytic>
</biblStruct>
</listBibl>
</div1>
</back>
</TEI>
<pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">BMC Bioinformatics</journal-id>
<journal-id journal-id-type="iso-abbrev">BMC Bioinformatics</journal-id>
<journal-title-group><journal-title>BMC Bioinformatics</journal-title>
</journal-title-group>
<issn pub-type="epub">1471-2105</issn>
<publisher><publisher-name>BioMed Central</publisher-name>
<publisher-loc>London</publisher-loc>
</publisher>
</journal-meta>
<article-meta><article-id pub-id-type="pmid">28292256</article-id>
<article-id pub-id-type="pmc">5351215</article-id>
<article-id pub-id-type="publisher-id">1527</article-id>
<article-id pub-id-type="doi">10.1186/s12859-017-1527-x</article-id>
<article-categories><subj-group subj-group-type="heading"><subject>Research Article</subject>
</subj-group>
</article-categories>
<title-group><article-title>A novel measure and significance testing in data analysis of cell image segmentation</article-title>
</title-group>
<contrib-group><contrib contrib-type="author" corresp="yes"><contrib-id contrib-id-type="orcid">http://orcid.org/0000-0002-6340-2467</contrib-id>
<name><surname>Wu</surname>
<given-names>Jin Chu</given-names>
</name>
<address><email>jinchu.wu@nist.gov</email>
</address>
<xref ref-type="aff" rid="Aff1"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Halter</surname>
<given-names>Michael</given-names>
</name>
<address><email>michael.halter@nist.gov</email>
</address>
<xref ref-type="aff" rid="Aff1"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Kacker</surname>
<given-names>Raghu N.</given-names>
</name>
<address><email>raghu.kacker@nist.gov</email>
</address>
<xref ref-type="aff" rid="Aff1"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Elliott</surname>
<given-names>John T.</given-names>
</name>
<address><email>john.elliott@nist.gov</email>
</address>
<xref ref-type="aff" rid="Aff1"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Plant</surname>
<given-names>Anne L.</given-names>
</name>
<address><email>anne.plant@nist.gov</email>
</address>
<xref ref-type="aff" rid="Aff1"></xref>
</contrib>
<aff id="Aff1"><institution-wrap><institution-id institution-id-type="ISNI">000000012158463X</institution-id>
<institution-id institution-id-type="GRID">grid.94225.38</institution-id>
<institution></institution>
<institution>National Institute of Standards and Technology,</institution>
</institution-wrap>
Gaithersburg, MD 20899 USA</aff>
</contrib-group>
<pub-date pub-type="epub"><day>14</day>
<month>3</month>
<year>2017</year>
</pub-date>
<pub-date pub-type="pmc-release"><day>14</day>
<month>3</month>
<year>2017</year>
</pub-date>
<pub-date pub-type="collection"><year>2017</year>
</pub-date>
<volume>18</volume>
<elocation-id>168</elocation-id>
<history><date date-type="received"><day>11</day>
<month>8</month>
<year>2016</year>
</date>
<date date-type="accepted"><day>6</day>
<month>2</month>
<year>2017</year>
</date>
</history>
<permissions><copyright-statement>© The Author(s). 2017</copyright-statement>
<license license-type="OpenAccess"><license-p><bold>Open Access</bold>
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0/">http://creativecommons.org/licenses/by/4.0/</ext-link>
), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/publicdomain/zero/1.0/">http://creativecommons.org/publicdomain/zero/1.0/</ext-link>
) applies to the data made available in this article, unless otherwise stated.</license-p>
</license>
</permissions>
<abstract id="Abs1"><sec><title>Background</title>
<p>Cell image segmentation (CIS) is an essential part of quantitative imaging of biological cells. Designing a performance measure and conducting significance testing are critical for evaluating and comparing the CIS algorithms for image-based cell assays in cytometry. Many measures and methods have been proposed and implemented to evaluate segmentation methods. However, computing the standard errors (SE) of the measures and their correlation coefficient is not described, and thus the statistical significance of performance differences between CIS algorithms cannot be assessed.</p>
</sec>
<sec><title>Results</title>
<p>We propose the total error rate (TER), a novel performance measure for segmenting all cells in the supervised evaluation. The TER statistically aggregates all misclassification error rates (MER) by taking cell sizes as weights. The MERs are for segmenting each single cell in the population. The TER is fully supported by the pairwise comparisons of MERs using 106 manually segmented ground-truth cells with different sizes and seven CIS algorithms taken from ImageJ. Further, the SE and 95% confidence interval (CI) of TER are computed based on the SE of MER that is calculated using the bootstrap method. An algorithm for computing the correlation coefficient of TERs between two CIS algorithms is also provided. Hence, the 95% CI error bars can be used to classify CIS algorithms. The SEs of TERs and their correlation coefficient can be employed to conduct the hypothesis testing, while the CIs overlap, to determine the statistical significance of the performance differences between CIS algorithms.</p>
</sec>
<sec><title>Conclusions</title>
<p>A novel measure TER of CIS is proposed. The TER’s SEs and correlation coefficient are computed. Thereafter, CIS algorithms can be evaluated and compared statistically by conducting the significance testing.</p>
</sec>
</abstract>
<kwd-group xml:lang="en"><title>Keywords</title>
<kwd>Cell image segmentation</kwd>
<kwd>Cell assays</kwd>
<kwd>Performance measure</kwd>
<kwd>Misclassification error rate</kwd>
<kwd>Total error rate</kwd>
<kwd>Standard error</kwd>
<kwd>Confidence interval</kwd>
<kwd>Correlation coefficient</kwd>
<kwd>Significance testing</kwd>
<kwd>Bootstrap method</kwd>
</kwd-group>
<custom-meta-group><custom-meta><meta-name>issue-copyright-statement</meta-name>
<meta-value>© The Author(s) 2017</meta-value>
</custom-meta>
</custom-meta-group>
</article-meta>
</front>
<body><sec id="Sec1" sec-type="introduction"><title>Background</title>
<p>Cell image segmentation (CIS) is an essential part of quantitative imaging of biological cells, which is critical to fields such as high content screening, live cell tracking and analysis, and the analysis of subcellular structures [<xref ref-type="bibr" rid="CR1">1</xref>
–<xref ref-type="bibr" rid="CR3">3</xref>
]. Segmenting cells from fluorescent microscopy images for image-based cell assays in cytometry requires the design and development of algorithms that are optimized for a particular set of images. The performance of a CIS algorithm can affect the quantitative results derived from an image analysis pipeline.</p>
<p>In order to use the well-established statistical approach to evaluate and compare CIS algorithms [<xref ref-type="bibr" rid="CR4">4</xref>
] so that the statistical significance of the performance differences between CIS algorithms can be determined, besides designing a novel CIS performance measure, the standard error (SE) of the measure and the correlation coefficient of measures between two CIS algorithms must be solved. These three issues are all dealt with in this article.</p>
<p>In this study, only supervised evaluation is carried out. Cells segmented manually by experts are treated as the ground-truth (GT) cells, whereas cells segmented using an algorithm are named as the algorithm-detected (AD) cells. The set-theoretic relationship between a GT cell and its related AD cell, as shown in Fig. <xref rid="Fig1" ref-type="fig">1</xref>
, consists of three regions: 1) the intersection region, the pixels of the GT cell identified by the algorithm; 2) the false negative (FN) region, the pixels of the GT cell missed by the algorithm; 3) the false positive (FP) region, the pixels of the AD cell that are mistakenly picked up and do not belong to the GT cell.<fig id="Fig1"><label>Fig. 1</label>
<caption><p>A schematic diagram showing the set-theoretic relationship between a GT cell and an AD cell where the sizes of regions are shown in terms of pixel numbers</p>
</caption>
<graphic xlink:href="12859_2017_1527_Fig1_HTML" id="MO1"></graphic>
</fig>
</p>
<p>The numbers of pixels of the GT cell, the FN region, the AD cell, the FP region, and the intersection region are denoted by <italic>n</italic>
<sub>G</sub>
, <italic>n</italic>
<sub>g</sub>
, <italic>n</italic>
<sub>A</sub>
, <italic>n</italic>
<sub>a</sub>
, and <italic>n</italic>
<sub>I</sub>
, respectively, which are subject to the constraint condition <italic>n</italic>
<sub>G</sub>
– <italic>n</italic>
<sub>g</sub>
= <italic>n</italic>
<sub>A</sub>
– <italic>n</italic>
<sub>a</sub>
= <italic>n</italic>
<sub>I</sub>
. The FN rate is <italic>n</italic>
<sub>g</sub>
/ <italic>n</italic>
<sub>G</sub>
, and the FP rate equals <italic>n</italic>
<sub>a</sub>
/ <italic>n</italic>
<sub>A</sub>
. In this article, it is assumed that all AD cells are counted as one AD cell taken on the level of pixels if they are related to one GT cell; and all GT cells are treated as one GT cell taken on the level of pixels if they are associated with one AD cell.</p>
<p>Some CIS algorithms may perform better than others for cells with some specific characteristics. Many measures and methods have been proposed and implemented to evaluate the performance of segmentation methods, such as the Jaccard index, the Rand index, the Kappa statistic, and others as shown in the literature<xref ref-type="fn" rid="Fn1">1</xref>
[<xref ref-type="bibr" rid="CR5">5</xref>
–<xref ref-type="bibr" rid="CR9">9</xref>
]. However, computing the SEs of the measures and their correlation coefficient is not described, although the uncertainty of the Kappa statistic was computed only for very small sizes of samples [<xref ref-type="bibr" rid="CR10">10</xref>
].</p>
<p>In this article, it starts with defining the misclassification error rate (MER) for segmenting a single cell in a fluorescent microscopy image. Two MERs are discussed: the average MER <italic>r</italic>
<sub>a</sub>
that is an arithmetic mean of the FN rate and the FP rate, and the weighted MER <italic>r</italic>
<sub>w</sub>
that is a weighted sum of these two rates using themselves as weights. The latter is more conservative than the former. Thus, the weighted MER <italic>r</italic>
<sub>w</sub>
is recommended. Then, the total error rate (TER), which is a novel performance measure for segmenting all cells, is defined to be a weighted sum of all MERs, and thus statistically aggregates all MERs. The weight is the size of a GT cell divided by the total size of all GT cells in the population. Hence, the penalties on the result for an algorithm are higher if larger GT cells are not segmented correctly.</p>
<p>Weight is widely employed in scientific research. In our research, as stated above, error rates are used as weight in the definition of the weighted MER, and the sizes of GT cells are used as weight while defining the TER that is a consequence of using the formula of the total probability in statistics (see section “<xref rid="Sec4" ref-type="sec">The TER for segmenting all cells</xref>
”). In different applications, weight may have different concepts, for instance, in Ref. [<xref ref-type="bibr" rid="CR11">11</xref>
].</p>
<p>Many factors can affect how accurately a CIS algorithm detects the boundary of a cell. The cell size is one major factor. Many approaches have taken account of the size factor, but in different contexts such as the unsupervised objective evaluation methods [<xref ref-type="bibr" rid="CR12">12</xref>
].</p>
<p>The TER is supported by the pairwise comparisons of MERs using 106 manually segmented GT cells with different sizes and seven CIS algorithms obtained from ImageJ [<xref ref-type="bibr" rid="CR13">13</xref>
]. The CIS algorithms are IJ_Huang, IJ_RenyiEntropy, IJ_Li, IJ_MaxEntropy, IJ_Intermodes, IJ_Minimum and IJ_Triangle, numbered by 1 through 7 according to their performance levels in descending order.</p>
<p>The 106 cells were taken from the A10 rat smooth muscle cell line. The raw image data and manual segmentation mask data are stored at the National Institute of Standards and Technology Semantics for Biological Data Resource: Cell Image Database [<xref ref-type="bibr" rid="CR14">14</xref>
]. The imaged cells were stained TxRed c2 maleimide (Invitrogen) (5 mg/mL in DMSO stock) which labels sulfhydryl groups present on cellular proteins. Fluorescence images were acquired with an Olympus IX71 inverted microscope (Center Valley, PA) equipped with an automated stage (Ludl, Hawthorne, NY), automated filter wheels (Ludl), a Xe arc lamp fluorescence excitation source, a 10 x ApoPlan 0.4 NA objective (Olympus), and a CoolSNAP HQ CCD camera (Roper Scientific, Tucson, AZ). The filter conditions for imaging the TxRed stained cells were a 555 nm notch excitation (PN# S555_25x, Chroma Technologies, Brattleboro, VT) and a 630 nm notch emission filter (PN#S630_60m). In Fig. <xref rid="Fig2" ref-type="fig">2</xref>
, nine fluorescent microscopy images illustrate the data used, where the cell sizes vary (concerning details of cell sizes, see section <xref rid="Sec12" ref-type="sec">Results</xref>
below).<fig id="Fig2"><label>Fig. 2</label>
<caption><p>Nine fluorescent microscopy images of representative A10 rat smooth muscle cells selected from 106 manually segmented cells</p>
</caption>
<graphic xlink:href="12859_2017_1527_Fig2_HTML" id="MO2"></graphic>
</fig>
</p>
<p>The results derived from the TER are also consistent with the qualitative observations from the histograms of MER (see section “<xref rid="Sec14" ref-type="sec">The TER and its SE and CI</xref>
”). More importantly, the TER method is more effective than the bivariate approach using the scatter plot of the FN and FP rates, and the approach using cumulative distribution function (CDF) of MER. It is challenging to compare the performance of two CIS algorithms while the two scatter plots overlap or the two CDF curves of MER cross each other.</p>
<p>Then, the issue is how to estimate the SE of the TER. In this article, the SE and 95% confidence interval (CI) of the TER for CIS algorithms are computed based on the SE of MER. The calculation of the SE of MER was accomplished by using the nonparametric bootstrap method under the constraint condition shown above.</p>
<p>To do so, dummy scores are assigned to pixels in different regions. For a GT cell, Score 2 is assigned to all n<sub>I</sub>
pixels in the intersection region and Score 0 is assigned to all n<sub>g</sub>
pixels in the FN region. For its related AD cell, Score 0 is assigned to all n<sub>I</sub>
pixels in the intersection region and Score 2 is assigned to all n<sub>a</sub>
pixels in the FP region. And a threshold is assigned to be 1.</p>
<p>As a result, the score distributions of a GT cell and of its related AD cell are similar to those in the receiver operating characteristic (ROC) analysis [<xref ref-type="bibr" rid="CR15">15</xref>
–<xref ref-type="bibr" rid="CR20">20</xref>
]. Indeed, the FN rate and the FP rate with respect to the threshold 1 in the CIS are exactly the same as the cumulative probabilities of type I error and type II error in ROC analysis, respectively. And in ROC analysis, the SEs of statistics of interest can be computed using the nonparametric bootstrap method.</p>
<p>Our bootstrap scheme is carried out under the constraint condition <italic>n</italic>
<sub>G</sub>
– <italic>n</italic>
<sub>g</sub>
= <italic>n</italic>
<sub>A</sub>
– <italic>n</italic>
<sub>a</sub>
= <italic>n</italic>
<sub>I</sub>
during bootstrap resampling, which is particular required for the CIS in the supervised evaluation involving GT cells and AD cells, as depicted in Fig. <xref rid="Fig1" ref-type="fig">1</xref>
. In the meantime, the stochastic nature of the bootstrap method is explored in this article.</p>
<p>The bootstrap is applied in many areas such as evaluating stability of clusters [<xref ref-type="bibr" rid="CR21">21</xref>
]. For different applications with different statistics of interest under different circumstances, there are many different bootstrap schemes about how to resample the original data. Due to our cell sizes, nonetheless, it is computationally prohibitive to generate the exact bootstrap distribution formed by all possible bootstrap replications of the statistic of interest [<xref ref-type="bibr" rid="CR10">10</xref>
].</p>
<p>In this article, moreover, it demonstrates how the correlation coefficient of TERs between two CIS algorithms is computed using the synchronized resampling algorithm. Without the correlation coefficient of TERs, the two-algorithm hypothesis testing cannot be conducted [<xref ref-type="bibr" rid="CR4">4</xref>
].</p>
<p>All these are very useful and important in the practice of CIS. The error bars of the TER displaying the 95% CI can be used to determine whether the difference between the performance level of a CIS algorithm and a hypothesized value is statistically significant in evaluation of CIS algorithms. This is related to the one-algorithm hypothesis testing, which can simply be judged by observing whether the 95% CI of the TER contains, below, or above the hypothesized value [<xref ref-type="bibr" rid="CR17">17</xref>
].</p>
<p>The error bars of the TER can also be used to classify CIS algorithms into different classes in terms of performance accuracies in comparison of CIS algorithms. When the CIs overlap within the same class, because the SE of TER and the correlation coefficient can be computed, the two-algorithm hypothesis testing can be conducted to determine the statistical significance of the performance difference between two CIS algorithms. In this article, only the two-algorithm hypothesis testing will be detailed.</p>
</sec>
<sec id="Sec2"><title>Methods</title>
<sec id="Sec3"><title>The MER for segmenting a single cell</title>
<p>The design of a novel performance measure in the CIS data analysis starts with defining the MER for identifying a single cell in a fluorescent image in the supervised evaluation. As stated above, the numbers of pixels, <italic>n</italic>
<sub>G</sub>
, <italic>n</italic>
<sub>g</sub>
, <italic>n</italic>
<sub>A</sub>
, <italic>n</italic>
<sub>a</sub>
, and <italic>n</italic>
<sub>I</sub>
, must satisfy the constraint condition,<disp-formula id="Equ1"><label>1</label>
<alternatives><tex-math id="M1">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ {n}_{\mathrm{G}} - {n}_{\mathrm{g}} = {n}_{\mathrm{A}} - {n}_{\mathrm{a}} = {n}_{\mathrm{I}}\ . $$\end{document}</tex-math>
<mml:math id="M2"><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">G</mml:mi>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">g</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">I</mml:mi>
</mml:msub>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mtext>.</mml:mtext>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ1.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>The FN rate <italic>r</italic>
<sub>fn</sub>
and the FP rate <italic>r</italic>
<sub>fp</sub>
are<disp-formula id="Equ2"><label>2</label>
<alternatives><tex-math id="M3">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ {r}_{\mathrm{fn}} = \frac{n_{\mathrm{g}}}{n_{\mathrm{G}}}\ \mathrm{and}\ {r}_{\mathrm{fp}} = \frac{n_{\mathrm{a}}}{n_{\mathrm{A}}}\ . $$\end{document}</tex-math>
<mml:math id="M4"><mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">g</mml:mi>
</mml:msub>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">G</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">and</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mtext>.</mml:mtext>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ2.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>Several MERs can be defined in terms of the FN rate <italic>r</italic>
<sub>fn</sub>
and the FP rate <italic>r</italic>
<sub>fp</sub>
. Besides “simplicity and ease of understanding” [<xref ref-type="bibr" rid="CR22">22</xref>
], conservativeness is also a criterion for defining MER in this article. Hence, two MERs are discussed as follows,<disp-formula id="Equ3"><label>3</label>
<alternatives><tex-math id="M5">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \begin{array}{l}{r}_{\mathrm{a}} = \frac{r_{\mathrm{fn}}+{r}_{\mathrm{fp}}}{2}, \\ {}{r}_{\mathrm{w}} = \frac{r_{\mathrm{fn}}^2 + {r}_{\mathrm{fp}}^2}{r_{\mathrm{fn}}+{r}_{\mathrm{fp}}}\ .\end{array} $$\end{document}</tex-math>
<mml:math id="M6"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:msub><mml:mi>r</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>,</mml:mo>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:msub><mml:mi>r</mml:mi>
<mml:mi mathvariant="normal">w</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:msubsup><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
</mml:mrow>
<mml:mrow><mml:mn>2</mml:mn>
</mml:mrow>
</mml:msubsup>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>+</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msubsup><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
</mml:mrow>
<mml:mrow><mml:mn>2</mml:mn>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mrow><mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mtext>.</mml:mtext>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ3.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>The average MER <italic>r</italic>
<sub>a</sub>
is an arithmetic mean of <italic>r</italic>
<sub>fn</sub>
and <italic>r</italic>
<sub>fp</sub>
, and the weighted MER <italic>r</italic>
<sub>w</sub>
is the one using <italic>r</italic>
<sub>fn</sub>
and <italic>r</italic>
<sub>fp</sub>
themselves as weight so that the larger error rate pays more penalties. As <italic>r</italic>
<sub>fn</sub>
and <italic>r</italic>
<sub>fp</sub>
approach to zero, <italic>r</italic>
<sub>w</sub>
goes to zero as well. Both <italic>r</italic>
<sub>a</sub>
and <italic>r</italic>
<sub>w</sub>
vary in the region [0, 1]: 0 stands for the best segmentation when an AD cell is identical to the related GT cell, and 1 means the worst classification when an AD cell and the associated GT cell are disjoint.</p>
<p>First, it is trivial to prove that the arithmetic mean of the FN rate <italic>r</italic>
<sub>fn</sub>
and the FP rate <italic>r</italic>
<sub>fp</sub>
is greater than or equal to the geometric mean <inline-formula id="IEq1"><alternatives><tex-math id="M7">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \sqrt{r_{\mathrm{fn}}{r}_{\mathrm{fp}}} $$\end{document}</tex-math>
<mml:math id="M8"><mml:msqrt><mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
</mml:mrow>
</mml:msub>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
</mml:mrow>
</mml:msub>
</mml:msqrt>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq1.gif"></inline-graphic>
</alternatives>
</inline-formula>
, which is subsequently greater than or equal to the harmonic mean 2<italic>r</italic>
<sub>fn</sub>
<italic>r</italic>
<sub>fp</sub>
/ (<italic>r</italic>
<sub>fn</sub>
+ <italic>r</italic>
<sub>fp</sub>
) [<xref ref-type="bibr" rid="CR23">23</xref>
]. These three means are all equal if and only if <italic>r</italic>
<sub>fn</sub>
= <italic>r</italic>
<sub>fp</sub>
. So, the arithmetic mean leads to more conservative (i.e. larger) estimates for the error rates.</p>
<p>Further, when either FN rate r<sub>fn</sub>
or FP rate r<sub>fp</sub>
approaches zero, both geometric and harmonic means go to zero, which indicates perfect segmentation in the supervised evaluation, no matter how much the other error rate is. Under such circumstances, however, the arithmetic mean approaches half of the other error rate.</p>
<p>Second, the weighted MER <italic>r</italic>
<sub>w</sub>
is compared with the average MER <italic>r</italic>
<sub>a</sub>
. Both of them are simple and easy to understand. However, as stated above, the weighted MER <italic>r</italic>
<sub>w</sub>
does penalize errors because of using the error rate as weight [<xref ref-type="bibr" rid="CR22">22</xref>
]. Moreover, it is trivial to prove from Eq. (3) that <italic>r</italic>
<sub>w</sub>
= <italic>r</italic>
<sub>a</sub>
if and only if <italic>r</italic>
<sub>fn</sub>
= <italic>r</italic>
<sub>fp</sub>
; otherwise, <italic>r</italic>
<sub>w</sub>
> <italic>r</italic>
<sub>a</sub>
. This can also be seen in Fig. <xref rid="Fig3" ref-type="fig">3</xref>
, where <italic>r</italic>
<sub>a</sub>
is a plane in green and <italic>r</italic>
<sub>w</sub>
is a surface in red as functions of <italic>r</italic>
<sub>fn</sub>
and <italic>r</italic>
<sub>fp</sub>
. The red surface is above the green plane except they are tangent along a straight line in blue. In other words, the weighted MER <italic>r</italic>
<sub>w</sub>
is a more conservative measure than the average MER <italic>r</italic>
<sub>a</sub>
.<fig id="Fig3"><label>Fig. 3</label>
<caption><p>The average MER <italic>r</italic>
<sub>a</sub>
is a plane (<italic>green</italic>
) and the weighted MER <italic>r</italic>
<sub>w</sub>
is a surface (<italic>red</italic>
) with respect to the FN rate <italic>r</italic>
<sub>fn</sub>
and the FP rate <italic>r</italic>
<sub>fp</sub>
. They are tangent along a straight line (<italic>blue</italic>
)</p>
</caption>
<graphic xlink:href="12859_2017_1527_Fig3_HTML" id="MO3"></graphic>
</fig>
</p>
<p>If an algorithm segments a small GT cell completely with a relatively very large AD cell, then <italic>r</italic>
<sub>fn</sub>
= 0 and <italic>r</italic>
<sub>fp</sub>
→ 1. If an algorithm detects a large GT cell with a relatively very small AD cell located completely inside the GT cell, then <italic>r</italic>
<sub>fp</sub>
= 0 and <italic>r</italic>
<sub>fn</sub>
→ 1. They all imply that <italic>r</italic>
<sub>w</sub>
→ 1 but <italic>r</italic>
<sub>a</sub>
→ 1/2 due to Eq. (3). These two cases can also be seen from Fig. <xref rid="Fig3" ref-type="fig">3</xref>
. Indeed, under these two circumstances, the MER should be much larger than 1/2 and close to 1. It indicates that the weighted MER <italic>r</italic>
<sub>w</sub>
can deal with these special cases better than the average MER <italic>r</italic>
<sub>a</sub>
, although in reality such special cases occur quite rarely.</p>
<p>Both <italic>r</italic>
<sub>w</sub>
and <italic>r</italic>
<sub>a</sub>
can be expressed as functions of the size of the intersection region <italic>n</italic>
<sub>I</sub>
so that a simulation can be carried out. The former is a linear function with negative slope. The latter is a more complicated function that decreases first and then increases as <italic>n</italic>
<sub>I</sub>
increases if <italic>n</italic>
<sub>G</sub>
≠ <italic>n</italic>
<sub>A</sub>
; but is the same function as the former if <italic>n</italic>
<sub>G</sub>
= <italic>n</italic>
<sub>A</sub>
. Both functions are symmetric with respect to <italic>n</italic>
<sub>G</sub>
and <italic>n</italic>
<sub>A</sub>
. All these imply that <italic>r</italic>
<sub>w</sub>
and <italic>r</italic>
<sub>a</sub>
behave differently when <italic>n</italic>
<sub>I</sub>
varies. That is, when an AD cell approaches to the related GT cell, <italic>r</italic>
<sub>w</sub>
decreases first and then increases if <italic>n</italic>
<sub>G</sub>
≠ <italic>n</italic>
<sub>A</sub>
, but <italic>r</italic>
<sub>a</sub>
always decreases. One may ask: Why cannot a CIS algorithm segment a GT cell completely when the AD cell is getting so close to it?</p>
<p>Based on these analyses, the weighted MER <italic>r</italic>
<sub>w</sub>
rather than the average MER <italic>r</italic>
<sub>a</sub>
is recommended. Thus, in this article, only those results computed using the weighted MER <italic>r</italic>
<sub>w</sub>
will be shown. Nonetheless, as far as computational results are concerned, qualitatively speaking, there is not too much discrepancy between the two MERs. These will be mentioned in the following text.</p>
<p>Figure <xref rid="Fig4" ref-type="fig">4</xref>
shows the histograms of the weighted MERs generated using Algorithms 1, 2, and 3 to segment 106 cells (see section <xref rid="Sec12" ref-type="sec">Results</xref>
). These three histograms overlap each other. However, they shift towards larger MER from Algorithm 1 to 3, suggesting that Algorithm 1 may be better than Algorithm 2 that in turn may be better than Algorithm 3. If the average MERs are employed, the relationship of the three histograms remains the same.<fig id="Fig4"><label>Fig. 4</label>
<caption><p>Histograms of the weighted MERs generated using Algorithms 1 (<bold>a</bold>
), 2 (<bold>b</bold>
), and 3 (<bold>c</bold>
)</p>
</caption>
<graphic xlink:href="12859_2017_1527_Fig4_HTML" id="MO4"></graphic>
</fig>
</p>
</sec>
<sec id="Sec4"><title>The TER for segmenting all cells</title>
<p>As pointed out in section <xref rid="Sec1" ref-type="sec">Background</xref>
, in this article, it is assumed that all AD cells are counted as one AD cell taken on the level of pixels if they are related to one GT cell; and all GT cells are treated as one GT cell taken on the level of pixels if they are associated with one AD cell. Hence, generally speaking, segmenting a cell in fluorescent microscopy images is an exclusive event with respect to detecting other cells. Then, to measure the performance level of a CIS algorithm, based on the formula of the total probability in statistics [<xref ref-type="bibr" rid="CR4">4</xref>
, <xref ref-type="bibr" rid="CR24">24</xref>
, <xref ref-type="bibr" rid="CR25">25</xref>
], the TER <italic>ε</italic>
is defined to be a weighted sum of all MERs,<disp-formula id="Equ4"><label>4</label>
<alternatives><tex-math id="M9">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \begin{array}{l}\varepsilon \equiv \Pr (CIS) = {\displaystyle \sum_{i=1}^N} \Pr \left( CIS\ \Big|{C}_i\right)\ \Pr \left({C}_i\right)\ \\ {}\kern5.25em = {\displaystyle \sum_{i=1}^N} ME{R}_i \times \frac{S_i}{{\displaystyle {\sum}_{j=1}^N}{S}_j}, \end{array} $$\end{document}</tex-math>
<mml:math id="M10"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:mi>ε</mml:mi>
<mml:mtext>≡</mml:mtext>
<mml:mo>Pr</mml:mo>
<mml:mfenced close=")" open="("><mml:mrow><mml:mi>C</mml:mi>
<mml:mi>I</mml:mi>
<mml:mi>S</mml:mi>
</mml:mrow>
</mml:mfenced>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mstyle displaystyle="true"><mml:munderover><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mi>N</mml:mi>
</mml:munderover>
</mml:mstyle>
<mml:mo>Pr</mml:mo>
<mml:mfenced close=")" open="("><mml:mrow><mml:mi>C</mml:mi>
<mml:mi>I</mml:mi>
<mml:mi>S</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo stretchy="true">|</mml:mo>
<mml:msub><mml:mi>C</mml:mi>
<mml:mi>i</mml:mi>
</mml:msub>
</mml:mrow>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>Pr</mml:mo>
<mml:mfenced close=")" open="("><mml:msub><mml:mi>C</mml:mi>
<mml:mi>i</mml:mi>
</mml:msub>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mspace width="5.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mstyle displaystyle="true"><mml:munderover><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mi>N</mml:mi>
</mml:munderover>
</mml:mstyle>
<mml:mi mathvariant="italic">ME</mml:mi>
<mml:msub><mml:mi>R</mml:mi>
<mml:mi>i</mml:mi>
</mml:msub>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>×</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:msub><mml:mi>S</mml:mi>
<mml:mi>i</mml:mi>
</mml:msub>
<mml:mrow><mml:mstyle displaystyle="true"><mml:msubsup><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>j</mml:mi>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow><mml:mi>N</mml:mi>
</mml:mrow>
</mml:msubsup>
</mml:mstyle>
<mml:msub><mml:mi>S</mml:mi>
<mml:mi>j</mml:mi>
</mml:msub>
</mml:mrow>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mtext>,</mml:mtext>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ4.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>where <italic>N</italic>
is the total number of GT cells, Pr(<italic>CIS</italic>
) stands for the total probability of making misclassification errors while using an algorithm to detect all cells in a fluorescent image, the conditional probability Pr(<italic>CIS</italic>
| <italic>C</italic>
<sub>i</sub>
) means the MER while segmenting the i-th GT cell in the image which is denoted by MER<sub>i</sub>
, and Pr(<italic>C</italic>
<sub>i</sub>
) is the probability of the occurrence of the i-th GT cell that is assumed to be the ratio of the size of the i-th GT cell <italic>S</italic>
<sub>i</sub>
to the total sizes of all GT cells. Hence, the TER ε statistically aggregates all cells’ MERs,</p>
<p>It can be proven that the TER <italic>ε</italic>
varies in the region [0, 1], where 0 stands for the best performance of the algorithm and 1 means the worst performance. As shown in Eq. (4), the cell sizes are used as weights. So, it can ensure that it penalizes errors and the penalties for misclassifying cells are proportional to the sizes of cells [<xref ref-type="bibr" rid="CR22">22</xref>
].</p>
</sec>
<sec id="Sec5"><title>The SE and 95% CI of TER</title>
<p>First, the SE of MER is computed using a bootstrap method. Second, based on that, the SE and 95% CI of TER are calculated. Third, the variation of the SE of TER is explored due to the stochastic nature of the bootstrap approach.</p>
<sec id="Sec6"><title>The SE of MER for segmenting a single cell</title>
<p>The MER for segmenting a single GT cell consists of the FN rate and the FP rate, and these two rates are formed by the numbers of pixels in different regions as shown from Eq. (1) to Eq. (3). Based on the assignment of dummy Scores 0 and 2 described in section <xref rid="Sec1" ref-type="sec">Background</xref>
, the score set for a GT cell is expressed as,<disp-formula id="Equ5"><label>5</label>
<alternatives><tex-math id="M11">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \mathbf{G} = \left\{{\mathrm{g}}_{\mathrm{i}}=0\left|\ \mathrm{i}=1, \dots, {n}_g;{\mathrm{g}}_{\mathrm{i}} = 2\ \right|\mathrm{i}={n}_{\mathrm{g}}+1, \dots, {n}_{\mathrm{G}}\right\}, $$\end{document}</tex-math>
<mml:math id="M12"><mml:mi mathvariant="bold">G</mml:mi>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfenced close="}" open="{"><mml:mrow><mml:msub><mml:mi mathvariant="normal">g</mml:mi>
<mml:mi mathvariant="normal">i</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mn>0</mml:mn>
<mml:mfenced close="|" open="|"><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">i</mml:mi>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>…</mml:mo>
<mml:mo>,</mml:mo>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi>g</mml:mi>
</mml:msub>
<mml:mo>;</mml:mo>
<mml:msub><mml:mi mathvariant="normal">g</mml:mi>
<mml:mi mathvariant="normal">i</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mn>2</mml:mn>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
<mml:mi mathvariant="normal">i</mml:mi>
<mml:mo>=</mml:mo>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">g</mml:mi>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>…</mml:mo>
<mml:mo>,</mml:mo>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">G</mml:mi>
</mml:msub>
</mml:mrow>
</mml:mfenced>
<mml:mtext>,</mml:mtext>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ5.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
and the score set for its related AD cell is denoted as,<disp-formula id="Equ6"><label>6</label>
<alternatives><tex-math id="M13">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \mathbf{A} = \left\{{\mathrm{a}}_{\mathrm{i}}=2\ \left|\ \mathrm{i}=1, \dots, {n}_{\mathrm{a}};\ {\mathrm{a}}_{\mathrm{i}}=0\ \right|\mathrm{i}={n}_{\mathrm{a}} + 1, \dots, {n}_{\mathrm{A}}\right\}, $$\end{document}</tex-math>
<mml:math id="M14"><mml:mi mathvariant="bold">A</mml:mi>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfenced close="}" open="{"><mml:mrow><mml:msub><mml:mi mathvariant="normal">a</mml:mi>
<mml:mi mathvariant="normal">i</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mn>2</mml:mn>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfenced close="|" open="|"><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">i</mml:mi>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>…</mml:mo>
<mml:mo>,</mml:mo>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:mo>;</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi mathvariant="normal">a</mml:mi>
<mml:mi mathvariant="normal">i</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mn>0</mml:mn>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
<mml:mi mathvariant="normal">i</mml:mi>
<mml:mo>=</mml:mo>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>…</mml:mo>
<mml:mo>,</mml:mo>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
</mml:mrow>
</mml:mfenced>
<mml:mtext>,</mml:mtext>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ6.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
where the constraint condition Eq. (1) must hold true.</p>
<p>There are five possibilities regarding the set-theoretic relationship between a GT cell and its associated AD cell: 1. the two cells are disjoint, 2. they are completely overlapped, 3. the GT cell completely contains the AD cell, 4. the AD cell completely contains the GT cell, 5. they are partially overlapped. Case 5 occurs most often in this study.</p>
<p>In the following, the bootstrap algorithm for computing the SE of MER is presented in a way to deal with Cases 4 and 5, in which both <italic>n</italic>
<sub>a</sub>
and <italic>n</italic>
<sub>I</sub>
= <italic>n</italic>
<sub>A</sub>
– <italic>n</italic>
<sub>a</sub>
are positive. Thus, the bootstrap random resampling with replacement (WR) can be legitimately applied to the score set of an AD cell in Eq. (6) [<xref ref-type="bibr" rid="CR15">15</xref>
–<xref ref-type="bibr" rid="CR20">20</xref>
]. Here is the nonparametric one-sample bootstrap algorithm of computing the SÊ of MER for segmenting a single cell.</p>
<p><graphic position="anchor" xlink:href="12859_2017_1527_Figa_HTML" id="MO5"></graphic>
</p>
<p>where M is the number of bootstrap replications. As shown from Step 1 to 10, this algorithm runs M times. In the i-th iteration, there is an endless <bold>while</bold>
loop from Step 2 to 8. In this loop, <italic>n</italic>
<sub>A</sub>
scores are randomly selected WR from the original score set <bold>A</bold>
in Eq. (6) to form a new score set <bold>A'</bold>
<sub>i</sub>
, which contains <italic>n</italic>
'<sub>a i</sub>
Score 2 forming a new FP region and <italic>n</italic>
<sub>A</sub>
- <italic>n</italic>
'<sub>a i</sub>
Score 0 forming a new intersection region, as shown in Step 3. Then, the constraint condition Eq. (1) needs to be checked. If the size of the new intersection region is less than or equal to the size of the GT cell in Step 4, the size of the new FN region, <italic>n</italic>
'<sub>g i</sub>
, is determined in Step 5 and the <bold>while</bold>
loop breaks in Step 6; otherwise, the endless <bold>while</bold>
loop continues.</p>
<p>After the <bold>while</bold>
loop breaks, the i-th estimated MÊR<sub>i</sub>
can be obtained in Step 9 from the new sizes of FP region and FN region, <italic>n</italic>
'<sub>a i</sub>
and <italic>n</italic>
'<sub>g i</sub>
, and the original sizes of the AC cell and the GT cell, <italic>n</italic>
<sub>A</sub>
and <italic>n</italic>
<sub>G</sub>
, using Eqs. (2) and (3). Finally, after M iterations, in Step 11, a bootstrap distribution is formed by the bootstrap replications of the MER, i.e., {MÊR<sub>i</sub>
| i = 1, …, M}, and then the standard error SÊ(MER)<sub>B</sub>
can be estimated using the sample standard deviation of this distribution.</p>
<p>Algorithm I can be easily converted to handling Case 3 in which there is no pixel in the FP region, if the score set of the GT cell in Eq. (5) is resampled. So, the scores and sets related to an AD cell should be replaced by the scores and sets related to a GT cell in Step 3, and the statements in Steps 4 and 5 should be changed to “<bold>if</bold>
<italic>n</italic>
<sub>G</sub>
- <italic>n</italic>
'<sub>g i</sub>
≤ <italic>n</italic>
<sub>A</sub>
<bold>then</bold>
” and “<italic>n</italic>
'<sub>a i</sub>
= <italic>n</italic>
<sub>A</sub>
- (<italic>n</italic>
<sub>G</sub>
- <italic>n</italic>
'<sub>g i</sub>
)” accordingly.</p>
<p>In Case 1, both <italic>r</italic>
<sub>fn</sub>
and <italic>r</italic>
<sub>fp</sub>
equal 1 and thus both <italic>r</italic>
<sub>a</sub>
and <italic>r</italic>
<sub>w</sub>
are 1. In Case 2, both <italic>r</italic>
<sub>fn</sub>
and <italic>r</italic>
<sub>fp</sub>
equal 0 and thus both <italic>r</italic>
<sub>a</sub>
and <italic>r</italic>
<sub>w</sub>
are 0. In these two cases, the estimates of SEs of both <italic>r</italic>
<sub>a</sub>
and <italic>r</italic>
<sub>w</sub>
are assumed to be 0, meaning that there is no variation associated with such MERs. So, the output of Algorithm I for Cases 1 and 2 is assumed to be zero.</p>
<p>The remaining issue is to determine how many iterations this bootstrap algorithm needs to run in order to reduce the bootstrap variance and ensure the accuracy of the computation. The appropriate number M of the bootstrap replications was determined to be 2000 based on our empirical bootstrap variability studies in ROC analysis [<xref ref-type="bibr" rid="CR17">17</xref>
–<xref ref-type="bibr" rid="CR20">20</xref>
].</p>
</sec>
<sec id="Sec7"><title>The SE and 95% CI of TER for segmenting all cells</title>
<p>After the bootstrap estimated SÊ of MER for segmenting each GT cell is computed, assuming that detecting and segmenting different GT cells in fluorescent microscopy images are mutually independent, the estimated variance of the TER <italic>ε</italic>
for detecting all GT cells can be obtained based on Eq. (4),<disp-formula id="Equ7"><label>7</label>
<alternatives><tex-math id="M15">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \mathrm{V}\mathrm{a}\mathrm{r}\ \left(\varepsilon \right) = {\displaystyle \sum_{i=1}^N}{\left(\frac{S_i}{{\displaystyle {\sum}_{j=1}^N}{S}_j}\right)}^2 \times \mathrm{S}\widehat{\mathrm{E}}{\left(\mathrm{MER}\right)}_{\mathrm{B}\ i}^2 $$\end{document}</tex-math>
<mml:math id="M16"><mml:mi mathvariant="normal">V</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
<mml:mi mathvariant="normal">r</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfenced close=")" open="("><mml:mi>ε</mml:mi>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mstyle displaystyle="true"><mml:munderover><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mi>N</mml:mi>
</mml:munderover>
</mml:mstyle>
<mml:msup><mml:mfenced close=")" open="("><mml:mfrac><mml:msub><mml:mi>S</mml:mi>
<mml:mi>i</mml:mi>
</mml:msub>
<mml:mrow><mml:mstyle displaystyle="true"><mml:msubsup><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>j</mml:mi>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow><mml:mi>N</mml:mi>
</mml:mrow>
</mml:msubsup>
</mml:mstyle>
<mml:msub><mml:mi>S</mml:mi>
<mml:mi>j</mml:mi>
</mml:msub>
</mml:mrow>
</mml:mfrac>
</mml:mfenced>
<mml:mn>2</mml:mn>
</mml:msup>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>×</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">S</mml:mi>
<mml:mover accent="true"><mml:mi mathvariant="normal">E</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:msubsup><mml:mfenced close=")" open="("><mml:mrow><mml:mi mathvariant="normal">M</mml:mi>
<mml:mi mathvariant="normal">E</mml:mi>
<mml:mi mathvariant="normal">R</mml:mi>
</mml:mrow>
</mml:mfenced>
<mml:mrow><mml:mi mathvariant="normal">B</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:msubsup>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ7.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>where <italic>N</italic>
is the total number of cells, <italic>S</italic>
<sub>i</sub>
is the size of the i-th GT cell, and SÊ(MER)<sub>B i</sub>
stands for the bootstrap estimated SÊ of MER for segmenting the i-th GT cell.</p>
<p>Then, the estimated SÊ of the TER <italic>ε</italic>
is defined to be the square root of Var (<italic>ε</italic>
). Again from Eq. (4), generally speaking, if no independent random variable dominates the others, the distribution of the TER can be assumed to be approximately normal because of the central limit theorem [<xref ref-type="bibr" rid="CR26">26</xref>
]. Thereafter, the estimated 95% CÎ of the TER <italic>ε</italic>
can be obtained by adding and subtracting 1.96 times the estimated SÊ.</p>
</sec>
<sec id="Sec8"><title>The variation of the SE of TER</title>
<p>The nature of the bootstrap method is stochastic. Each execution of the bootstrap algorithm may result in different SÊs of MERs and thus different SÊs of a TER. It is necessary to investigate how much the estimated SÊ of the TER varies. Hence, a distribution of such estimates needs to be generated. Here is the algorithm to create such a distribution.</p>
<p><graphic position="anchor" xlink:href="12859_2017_1527_Figb_HTML" id="MO6"></graphic>
</p>
<p>where M is the number of bootstrap replications, N is the total number of cells, L is the number of the Monte Carlo iterations, and Step 4 is the <bold>while</bold>
loop in Algorithm I from Step 2 to 8.</p>
<p>From Step 3 to 7, Algorithm I is employed to compute the SÊ (MER)<sub>B</sub>
of an MER for segmenting a single GT cell. From Step 2 to 8, Algorithm I is used to compute SÊs of MERs for all N GT cells. Thus, at Step 9, an estimated SÊ (<italic>ε</italic>
)<sub>B</sub>
of the TER <italic>ε</italic>
for detecting all GT cells is calculated using Eq. (7).</p>
<p>Such a process is executed in L times from Step 1 to 10. After L iterations, at Step 11, L estimated SÊ (<italic>ε</italic>
)<sub>B i</sub>
of the TER <italic>ε</italic>
are generated and constitute a distribution. Thereafter, the estimated SÊ<sub>B</sub>
and the (1–α)100% CÎ (<inline-formula id="IEq2"><alternatives><tex-math id="M17">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ {\widehat{Q}}_B $$\end{document}</tex-math>
<mml:math id="M18"><mml:msub><mml:mover accent="true"><mml:mi>Q</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi>B</mml:mi>
</mml:msub>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq2.gif"></inline-graphic>
</alternatives>
</inline-formula>
(α/2), <inline-formula id="IEq3"><alternatives><tex-math id="M19">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ {\widehat{Q}}_B $$\end{document}</tex-math>
<mml:math id="M20"><mml:msub><mml:mover accent="true"><mml:mi>Q</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi>B</mml:mi>
</mml:msub>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq3.gif"></inline-graphic>
</alternatives>
</inline-formula>
(1–α/2)) at the significance level α of the distribution can be computed.</p>
<p>The estimated α/2 100% and (1–α/2) 100% quantiles of the distribution are calculated using the Definition 2 of quantile in Ref. [<xref ref-type="bibr" rid="CR27">27</xref>
]. That is, the sample quantile is obtained by inverting the empirical distribution function with averaging at discontinuities. If 95% CÎ is of interest, then α is set to be 0.05.</p>
<p>Finally, the number of the Monte Carlo iterations L needs to be determined in order to guarantee the accuracy of the Monte Carlo computation. Based on our previous studies, to create a stable distribution, it is enough that the repeated process described above be executed 500 times, i.e., L = 500 [<xref ref-type="bibr" rid="CR17">17</xref>
–<xref ref-type="bibr" rid="CR20">20</xref>
].</p>
</sec>
</sec>
<sec id="Sec9"><title>Significance testing</title>
<p>The two-algorithm hypothesis testing is carried out by the Z test, since the TER can be assumed to be normally distributed as pointed out above [<xref ref-type="bibr" rid="CR4">4</xref>
].</p>
<sec id="Sec10"><title>Two-algorithm hypothesis testing</title>
<p>Let T<sub>A</sub>
and T<sub>B</sub>
denote the TERs for CIS Algorithms A and B, respectively. Then, the null and alternative hypotheses are<disp-formula id="Equ8"><label>8</label>
<alternatives><tex-math id="M21">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \begin{array}{l}{H}_{\mathrm{o}}:{T}_{\mathrm{A}}={T}_{\mathrm{B}}\\ {}{H}_{\mathrm{a}}:{T}_{\mathrm{A}}\ne {T}_{\mathrm{B}}\end{array} $$\end{document}</tex-math>
<mml:math id="M22"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:msub><mml:mi>H</mml:mi>
<mml:mi mathvariant="normal">o</mml:mi>
</mml:msub>
<mml:mo>:</mml:mo>
<mml:msub><mml:mi>T</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:msub><mml:mi>T</mml:mi>
<mml:mi mathvariant="normal">B</mml:mi>
</mml:msub>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:msub><mml:mi>H</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:mo>:</mml:mo>
<mml:msub><mml:mi>T</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
<mml:mo>≠</mml:mo>
<mml:msub><mml:mi>T</mml:mi>
<mml:mi mathvariant="normal">B</mml:mi>
</mml:msub>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ8.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>Based on the normality assumption, the general Z statistic for two-algorithm hypothesis testing is<disp-formula id="Equ9"><label>9</label>
<alternatives><tex-math id="M23">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ Z=\frac{\widehat{T}\mathrm{A}\ \hbox{-}\ \widehat{T}\mathrm{B}}{\sqrt{{\mathrm{SE}}^2\left(\widehat{T}\mathrm{A}\right) + {\mathrm{SE}}^2\left(\widehat{T}\mathrm{B}\right)\ \hbox{-}\ 2\ \rho\ \mathrm{SE}\left(\widehat{T}\mathrm{A}\right)\ \mathrm{SE}\left(\widehat{T}\mathrm{B}\right)}} $$\end{document}</tex-math>
<mml:math id="M24"><mml:mi>Z</mml:mi>
<mml:mo>=</mml:mo>
<mml:mfrac><mml:mrow><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">A</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>‐</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">B</mml:mi>
</mml:mrow>
<mml:msqrt><mml:mrow><mml:msup><mml:mrow><mml:mi mathvariant="normal">S</mml:mi>
<mml:mi mathvariant="normal">E</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:msup>
<mml:mfenced close=")" open="("><mml:mrow><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:mrow>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>+</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msup><mml:mrow><mml:mi mathvariant="normal">S</mml:mi>
<mml:mi mathvariant="normal">E</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:msup>
<mml:mfenced close=")" open="("><mml:mrow><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">B</mml:mi>
</mml:mrow>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>‐</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mn>2</mml:mn>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>ρ</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">S</mml:mi>
<mml:mi mathvariant="normal">E</mml:mi>
<mml:mfenced close=")" open="("><mml:mrow><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:mrow>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">S</mml:mi>
<mml:mi mathvariant="normal">E</mml:mi>
<mml:mfenced close=")" open="("><mml:mrow><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">B</mml:mi>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
</mml:msqrt>
</mml:mfrac>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ9.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>where <inline-formula id="IEq4"><alternatives><tex-math id="M25">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{T}\mathrm{A} $$\end{document}</tex-math>
<mml:math id="M26"><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq4.gif"></inline-graphic>
</alternatives>
</inline-formula>
and <inline-formula id="IEq5"><alternatives><tex-math id="M27">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{T}\mathrm{B} $$\end{document}</tex-math>
<mml:math id="M28"><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">B</mml:mi>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq5.gif"></inline-graphic>
</alternatives>
</inline-formula>
are two estimated TERs, SE(<inline-formula id="IEq6"><alternatives><tex-math id="M29">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{T}\mathrm{A} $$\end{document}</tex-math>
<mml:math id="M30"><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq6.gif"></inline-graphic>
</alternatives>
</inline-formula>
) and SE(<inline-formula id="IEq7"><alternatives><tex-math id="M31">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{T}\mathrm{B} $$\end{document}</tex-math>
<mml:math id="M32"><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">B</mml:mi>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq7.gif"></inline-graphic>
</alternatives>
</inline-formula>
) stand for their SEs, respectively, and ρ is the correlation coefficient between <inline-formula id="IEq8"><alternatives><tex-math id="M33">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{T}\mathrm{A} $$\end{document}</tex-math>
<mml:math id="M34"><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq8.gif"></inline-graphic>
</alternatives>
</inline-formula>
and <inline-formula id="IEq9"><alternatives><tex-math id="M35">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{T}\mathrm{B} $$\end{document}</tex-math>
<mml:math id="M36"><mml:mover accent="true"><mml:mi>T</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi mathvariant="normal">B</mml:mi>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq9.gif"></inline-graphic>
</alternatives>
</inline-formula>
.</p>
</sec>
<sec id="Sec11"><title>An algorithm for computing the correlation coefficient</title>
<p>This algorithm is based on the synchronized resampling approach. The two algorithms segment the same set of GT cells. The MERs of two CIS algorithms due to segmenting cells with the same ordinal number in the set of GT cells co-vary. As a result, the two TERs of any two CIS algorithms are correlated. The tendency of obtaining higher or lower MERs for segmenting the same GT cell could be different. Therefore, the correlation between the TERs of any two CIS algorithms may be positive or negative.</p>
<p>Using the notations in Eqs. (5) and (6), a score set that an Algorithm A segments the i-th GT cell with size <italic>n</italic>
<sub>G i</sub>
in the set of <italic>N</italic>
GT cells and generates <italic>n</italic>
<sup>A</sup>
<sub>g i</sub>
, <italic>n</italic>
<sup>A</sup>
<sub>A i</sub>
, and <italic>n</italic>
<sup>A</sup>
<sub>a i</sub>
accordingly is denoted by<disp-formula id="Equ10"><label>10</label>
<alternatives><tex-math id="M37">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ {\mathbf{C}}^A = \left\{{n}_{\mathrm{G}\ \mathrm{i}},{n^{\mathrm{A}}}_{\mathrm{g}\ \mathrm{i}},{n^{\mathrm{A}}}_{\mathrm{A}\ \mathrm{i}},{n^{\mathrm{A}}}_{\mathrm{a}\ \mathrm{i}}\Big|\mathrm{i}=1, \dots, N\right\}, $$\end{document}</tex-math>
<mml:math id="M38"><mml:msup><mml:mi mathvariant="bold">C</mml:mi>
<mml:mi>A</mml:mi>
</mml:msup>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfenced close="}" open="{"><mml:mrow><mml:msub><mml:mi>n</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">G</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>,</mml:mo>
<mml:msub><mml:msup><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msup>
<mml:mrow><mml:mi mathvariant="normal">g</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>,</mml:mo>
<mml:msub><mml:msup><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msup>
<mml:mrow><mml:mi mathvariant="normal">A</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>,</mml:mo>
<mml:msub><mml:msup><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msup>
<mml:mrow><mml:mi mathvariant="normal">a</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi mathvariant="normal">i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo stretchy="true">|</mml:mo>
<mml:mi mathvariant="normal">i</mml:mi>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>…</mml:mo>
<mml:mo>,</mml:mo>
<mml:mi>N</mml:mi>
</mml:mrow>
</mml:mfenced>
<mml:mtext>,</mml:mtext>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ10.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
from which a TER can be computed using Eqs. (2) through (4). All CIS algorithms segment the same set of <italic>N</italic>
GT cells. Thus, the size of the i-th GT cell, i.e., n<sub>G i</sub>
, is the same for all CIS algorithms. This correlates TERs of different algorithms.</p>
<p>An algorithm for computing the correlation coefficient of the TERs for CIS Algorithms A and B is as follows.</p>
<p><graphic position="anchor" xlink:href="12859_2017_1527_Figc_HTML" id="MO7"></graphic>
</p>
<p>where <italic>s</italic>
<sup>A</sup>
<sub><italic>k</italic>
</sub>
, <italic>γ</italic>
<sup>A</sup>
<sub><italic>j</italic>
</sub>
, <italic>s</italic>
<sup>B</sup>
<sub><italic>k</italic>
</sub>
, and <italic>γ</italic>
<sup>B</sup>
<sub><italic>j</italic>
</sub>
are members of the score sets <bold>S</bold>
<sup>A</sup>
, <bold>Γ</bold>
<sup>A</sup>
, <bold>S</bold>
<sup>B</sup>
, and <bold>Γ</bold>
<sup>B</sup>
, respectively. Based on our bootstrap variability studies, the number of iterations M is set to be 2000 [<xref ref-type="bibr" rid="CR17">17</xref>
–<xref ref-type="bibr" rid="CR20">20</xref>
].</p>
<p>From Step 1 to 5, this algorithm runs M iterations. In Step 2 of the i-th iteration, the synchronized WR random resampling is carried out on the two score sets <bold>C</bold>
<sup>A</sup>
and <bold>C</bold>
<sup>B</sup>
of Algorithms A and B to generate two new score sets <bold>Θ</bold>
<sup>A</sup>
<sub><italic>i</italic>
</sub>
and <bold>Θ</bold>
<sup>B</sup>
<sub><italic>i</italic>
</sub>
.</p>
<p>From Step 1.1 to 1.7, during the resampling iterations, if a member with index k in <bold>S</bold>
<sup><bold>A</bold>
</sup>
is WR randomly selected, then the member with the same index k in <bold>S</bold>
<sup><bold>B</bold>
</sup>
is also selected. That is, a GT cell with the same ordinal number k in the set of <italic>N</italic>
GT cells is selected. Thus, such synchronized selections guarantee that all co-varying members in score sets between the two CIS algorithms are selected simultaneously. Hence, the correlation of the TERs between the two algorithms is preserved.</p>
<p>After resampling, in Step 3 (4), the <italic>i</italic>
-th estimated TER <inline-formula id="IEq10"><alternatives><tex-math id="M39">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ {\widehat{\varepsilon}}_i^A $$\end{document}</tex-math>
<mml:math id="M40"><mml:msubsup><mml:mover accent="true"><mml:mi>ε</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi>i</mml:mi>
<mml:mi>A</mml:mi>
</mml:msubsup>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq10.gif"></inline-graphic>
</alternatives>
</inline-formula>
(<inline-formula id="IEq11"><alternatives><tex-math id="M41">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ {\widehat{\varepsilon}}_i^B $$\end{document}</tex-math>
<mml:math id="M42"><mml:msubsup><mml:mover accent="true"><mml:mi>ε</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
<mml:mi>i</mml:mi>
<mml:mi>B</mml:mi>
</mml:msubsup>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq11.gif"></inline-graphic>
</alternatives>
</inline-formula>
) of Algorithm A (B) is computed from the new score set <bold>Θ</bold>
<sup>A</sup>
<sub><italic>i</italic>
</sub>
(<bold>Θ</bold>
<sup>B</sup>
<sub><italic>i</italic>
</sub>
). Finally in Step 6 after M iterations, the correlation coefficient <italic>ρ</italic>
<sup>AB</sup>
of the TERs of Algorithms A and B is computed from the two sets of correlated TERs.</p>
<p>A synchronized random resampling is involved here. Thus, this algorithm needs to run multiple times to reduce the computational fluctuation, if the <italic>p</italic>
-value is not considerably different from the critical values, such as 5%, 1%, etc. To be more conservative, in this article, the average out of 10 runs was taken to be the resultant correlation coefficient for significance testing.</p>
</sec>
</sec>
</sec>
<sec id="Sec12" sec-type="results"><title>Results</title>
<p>The dataset consisted of 106 cells with different sizes, which were manually segmented as GT cells. Figure <xref rid="Fig5" ref-type="fig">5</xref>
shows the histogram of the cell size. The sizes ranged from 647 up to 27,562 pixels with the mean size at 6062 pixels. The variation of cell sizes was quite large. Thus, the cell sizes must be taken into account while evaluating CIS algorithms.<fig id="Fig5"><label>Fig. 5</label>
<caption><p>The histogram of the sizes of all 106 GT cells</p>
</caption>
<graphic xlink:href="12859_2017_1527_Fig5_HTML" id="MO8"></graphic>
</fig>
</p>
<p>The seven CIS algorithms in ImageJ were implemented. As stated above, the estimated SÊs of MERs in Cases 1 and 2 are zero, which can lower the estimate of the SE of TER for detecting all GT cells. Only Algorithm 4 created one Case 1 and Algorithm 7 produced three; and no algorithm generated Case 2.</p>
<sec id="Sec13"><title>Pairwise comparisons to support the TER</title>
<p>Two CIS algorithms generate two weighted MERs while segmenting the same GT cell, and these two weighted MERs are compared. Table <xref rid="Tab1" ref-type="table">1</xref>
shows the relationship in terms of the numbers of “less than” (<), “greater than” (>), and “equal to” (=) between such two weighted MERs while using two algorithms to segment all 106 GT cells. For instance, comparing Algorithms 1 with 2, for 87 GT cells, the weighted MERs generated using Algorithm 1 are less than those created using Algorithm 2; and for only 19 GT cells, the “greater than” occurs otherwise. This indicates that the performance of Algorithm 1 is better than the performance of Algorithm 2.<table-wrap id="Tab1"><label>Table 1</label>
<caption><p>Comparisons of the weighted MERs generated using two algorithms for all 106 cells in terms of the numbers of inequalities and equalities</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th rowspan="2" colspan="2">Algorithm</th>
<th colspan="3">the number of</th>
</tr>
<tr><th><</th>
<th>></th>
<th>=</th>
</tr>
</thead>
<tbody><tr><td>1</td>
<td>2</td>
<td>87</td>
<td>19</td>
<td>0</td>
</tr>
<tr><td>2</td>
<td>3</td>
<td>57</td>
<td>49</td>
<td>0</td>
</tr>
<tr><td>3</td>
<td>4</td>
<td>68</td>
<td>38</td>
<td>0</td>
</tr>
<tr><td>4</td>
<td>5</td>
<td>59</td>
<td>47</td>
<td>0</td>
</tr>
<tr><td>5</td>
<td>6</td>
<td>101</td>
<td>5</td>
<td>0</td>
</tr>
<tr><td>6</td>
<td>7</td>
<td>79</td>
<td>27</td>
<td>0</td>
</tr>
</tbody>
</table>
</table-wrap>
</p>
<p>Further, in Table <xref rid="Tab1" ref-type="table">1</xref>
, the relationship of “better than” is transitive. For example, the performance of Algorithm 1 is also better than the performance of Algorithm 3, and so on. Indeed, while comparing Algorithms 1 with 3, there are 91 “<” and 15 “>”. As a result, the test of pairwise comparisons of MERs in this article was conducted between any two out of seven CIS algorithms. Table <xref rid="Tab1" ref-type="table">1</xref>
shows that the performance is degraded in the ascending order of the CIS algorithms.</p>
<p>If the average MERs are employed, qualitatively speaking, the relationship among these CIS algorithms in terms of numbers of “<”, “>”, and “=” stays the same.</p>
</sec>
<sec id="Sec14"><title>The TER and its SE and CI</title>
<p>Table <xref rid="Tab2" ref-type="table">2</xref>
shows the estimated TÊRs, SÊs (relative errors) and 95% CÎs of TERs for the seven CIS algorithms, when the weighted MERs are employed. The smaller the estimated TÊR ε is, the better the performance is. The order of the algorithms in Table <xref rid="Tab2" ref-type="table">2</xref>
is consistent with the one in Table <xref rid="Tab1" ref-type="table">1</xref>
. It indicates that the TER constructed on all MERs and using the cell sizes as weights is fully supported by the results derived directly from the pairwise-comparison test of MERs using 106 GT cells with different sizes and seven CIS algorithms taken from ImageJ.<table-wrap id="Tab2"><label>Table 2</label>
<caption><p>The estimated TÊRs, SÊs (relative errors) and 95% CÎs of TERs for the seven CIS algorithms, in which the weighted MERs are employed</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th>Alg.</th>
<th>TÊR</th>
<th>SÊ (relative error)</th>
<th>95% CÎ of TER</th>
</tr>
</thead>
<tbody><tr><td>1</td>
<td char="." align="char">0.057524</td>
<td>0.000893 (3.04%)</td>
<td>(0.055775, 0.059274)</td>
</tr>
<tr><td>2</td>
<td char="." align="char">0.066889</td>
<td>0.000093 (0.27%)</td>
<td>(0.066707, 0.067071)</td>
</tr>
<tr><td>3</td>
<td char="." align="char">0.089363</td>
<td>0.000674 (1.48%)</td>
<td>(0.088042, 0.090684)</td>
</tr>
<tr><td>4</td>
<td char="." align="char">0.105096</td>
<td>0.000061 (0.11%)</td>
<td>(0.104976, 0.105215)</td>
</tr>
<tr><td>5</td>
<td char="." align="char">0.171153</td>
<td>0.001721 (1.97%)</td>
<td>(0.167780, 0.174526)</td>
</tr>
<tr><td>6</td>
<td char="." align="char">0.173513</td>
<td>0.000868 (0.98%)</td>
<td>(0.171812, 0.175213)</td>
</tr>
<tr><td>7</td>
<td char="." align="char">0.224444</td>
<td>0.000095 (0.08%)</td>
<td>(0.224257, 0.224631)</td>
</tr>
</tbody>
</table>
</table-wrap>
</p>
<p>Moreover, regarding Algorithms 1, 2, and 3, their estimated TÊR ε shown in Table <xref rid="Tab2" ref-type="table">2</xref>
are qualitatively consistent with the observations in Fig. <xref rid="Fig4" ref-type="fig">4</xref>
, where the histograms of the weighted MERs for these three algorithms shift gradually towards larger MER.</p>
<p>The relative error of the TER can be defined as “1.96 × SÊ / TÊR”, where 1.96 is the Z score corresponding to 95% CI. Thus, the ranges of relative errors are between 0.08% and 3.04%. Most importantly, Fig. <xref rid="Fig6" ref-type="fig">6</xref>
shows the error bars of the TER displaying the 95% CÎs along with estimated TÊRs for six CIS algorithms, when the weighted MERs are employed. Algorithm 7 is not included due to large TÊR.<fig id="Fig6"><label>Fig. 6</label>
<caption><p>The error bars of TER displaying 95% CIs along with estimated TERs for six CIS algorithms, in which the weighted MERs are employed, with two criteria set at 0.08 and 0.14 that statistically classify CIS algorithms</p>
</caption>
<graphic xlink:href="12859_2017_1527_Fig6_HTML" id="MO9"></graphic>
</fig>
</p>
<p>If the average MERs are employed, the corresponding estimated TÊRs are smaller, which is consistent with what was discussed in sections “<xref rid="Sec3" ref-type="sec">The MER for segmenting a single cell</xref>
” and “<xref rid="Sec4" ref-type="sec">The TER for segmenting all cells</xref>
” (see section <xref rid="Sec17" ref-type="sec">Discussion</xref>
). But qualitatively speaking, except that the 95% CÎ of TER of Algorithm 1 contains the one of Algorithm 2, everything else stated here remains the same.</p>
</sec>
<sec id="Sec15"><title>The variation of the estimated SÊ of the TER</title>
<p>The nature of the bootstrap method is stochastic. Algorithm II was used to deal with this issue. Table <xref rid="Tab3" ref-type="table">3</xref>
shows the means, SÊs (relative errors), and 95% CÎs of the estimated SÊs of TÊRs for the seven CIS algorithms, when the weighted MERs are employed. The relative error is defined as “1.96 × SE / mean” to take account of all estimates occurring in the estimated 95% CÎ. All 95% CÎs are quite narrow. The largest relative error is 1.87%.<table-wrap id="Tab3"><label>Table 3</label>
<caption><p>The means, SÊs (relative errors), and 95% CÎs of the estimated SÊs of TERs for the seven CIS algorithms, in which the weighted MERs are employed</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th>Alg.</th>
<th>Mean</th>
<th>SÊ (relative error)</th>
<th>95% CÎ of SÊ of TER</th>
</tr>
</thead>
<tbody><tr><td>1</td>
<td char="." align="char">0.000903</td>
<td>0.000007 (1.47%)</td>
<td>(0.000890, 0.000916)</td>
</tr>
<tr><td>2</td>
<td char="." align="char">0.000093</td>
<td>0.000000 (0.57%)</td>
<td>(0.000092, 0.000093)</td>
</tr>
<tr><td>3</td>
<td char="." align="char">0.000668</td>
<td>0.000006 (1.87%)</td>
<td>(0.000657, 0.000682)</td>
</tr>
<tr><td>4</td>
<td char="." align="char">0.000061</td>
<td>0.000000 (0.99%)</td>
<td>(0.000060, 0.000061)</td>
</tr>
<tr><td>5</td>
<td char="." align="char">0.001712</td>
<td>0.000012 (1.36%)</td>
<td>(0.001689, 0.001735)</td>
</tr>
<tr><td>6</td>
<td char="." align="char">0.000874</td>
<td>0.000006 (1.36%)</td>
<td>(0.000863, 0.000886)</td>
</tr>
<tr><td>7</td>
<td char="." align="char">0.000096</td>
<td>0.000000 (0.97%)</td>
<td>(0.000095, 0.000097)</td>
</tr>
</tbody>
</table>
</table-wrap>
</p>
<p>Figure <xref rid="Fig7" ref-type="fig">7</xref>
shows the histograms of the estimated SÊs of TERs for CIS Algorithms 1 (blue), 3 (red), 5 (green), and 6 (gray), when the weighted MERs are employed. The histograms of other three algorithms are too narrow to draw. The widths of all distributions are very narrow, demonstrating that the results are quite stable.<fig id="Fig7"><label>Fig. 7</label>
<caption><p>The histograms of the estimated SÊs of TÊRs for four CIS Algorithms 1 (blue), 3 (red), 5 (green), and 6 (gray), in which the weighted MERs are employed. The black circle stands for the estimated mean of the distribution</p>
</caption>
<graphic xlink:href="12859_2017_1527_Fig7_HTML" id="MO10"></graphic>
</fig>
</p>
<p>Taking Algorithm 1 as an example whose TÊR in Table <xref rid="Tab2" ref-type="table">2</xref>
is the smallest and relative error is the largest, and using Algorithm 1’s estimated 95% CÎ of SÊs of TERs in Table <xref rid="Tab3" ref-type="table">3</xref>
, it can be calculated that the relative errors of TÊR may vary between 3.03% and 3.12%.</p>
<p>It is worth mentioning that in Table <xref rid="Tab2" ref-type="table">2</xref>
, all estimated SÊs of TERs were calculated by a random execution of the stochastic bootstrap method while computing the SÊs of MERs. However, they all correspondingly fall in the 95% CÎ of the estimated SÊs of TERs shown in Table <xref rid="Tab3" ref-type="table">3</xref>
.</p>
<p>Again, if the average MERs are employed, qualitatively speaking, nothing stated here is changed. For instance, the relative errors of TÊR for Algorithm 1 varies between 5.31% and 5.56%, which is also very narrow.</p>
</sec>
<sec id="Sec16"><title>Significance testing</title>
<p>CIS algorithms may be statistically classified into different classes in terms of performance accuracies using the error bars. This provides a basis for identifying algorithms that are quantitatively similar to one another. For instance, as shown in Fig. <xref rid="Fig6" ref-type="fig">6</xref>
, if the criteria of performance accuracies are set to be at 0.08 and 0.14, respectively, then Algorithms 1 and 2 are classified to be in the first class, Algorithms 3 and 4 are in the second class, and Algorithms 5 and 6 are in the third class. This is because their error bars, i.e., the 95% CÎs of TER, do not cross the criteria. Otherwise, the one-algorithm hypothesis testing needs to be taken. Certainly, the criteria are set depending on the circumstances being dealt with.</p>
<p>When two error bars do not overlap, for example, for Algorithms 1 and 2 as depicted in Fig. <xref rid="Fig6" ref-type="fig">6</xref>
, the performance level of the CIS algorithm corresponding to the lower error bar is better than the other one. When two error bars overlap, for example, for Algorithms 5 and 6, the two-algorithm hypothesis testing is necessary to determine the statistical significance of performance difference.</p>
<p>To demonstrate, the hypothesis testing is conducted on Algorithms 1 and 2, as well as on Algorithms 5 and 6. The corresponding correlation coefficients of TERs computed using Algorithm III are 0.215203, and 0.370554, respectively. Then, using the TERs and their SEs shown in Table <xref rid="Tab2" ref-type="table">2</xref>
, the Z-test two-tailed p-values are 0%, and 14.4% accordingly.</p>
<p>Using 5% as a critical p-value, these p-values show that the performance level of Algorithm 1 is better than the performance level of Algorithm 2, even though they are in the same first class. However, the difference in terms of performance accuracy between Algorithms 5 and 6 is not statistically significant.</p>
<p>If the average MERs are used, it is pointed out above that the 95% CÎ of TER of Algorithm 1 contains the one of Algorithm 2. This is consistent with the result of conducting the hypothesis testing. The p-value is 11.4%, which shows that the performance difference between Algorithms 1 and 2 is not statistically significant. This is the only difference qualitatively speaking between using the weighted MERs and the average MERs on our CIS datasets. Nonetheless, as analyzed in section “<xref rid="Sec3" ref-type="sec">The MER for segmenting a single cell</xref>
”, the weighted MER is recommended.</p>
</sec>
</sec>
<sec id="Sec17" sec-type="discussion"><title>Discussion</title>
<p>The MER for segmenting a single cell in the supervised evaluation can be defined in several ways. It is hard to find one without any disadvantages [<xref ref-type="bibr" rid="CR22">22</xref>
]. Simplicity, ease of understanding, penalizing errors, conservativeness, and dealing with special cases are the criteria of choosing MER in this article. Based on our analyses, the weighted MER <italic>r</italic>
<sub>w</sub>
is recommended. Certainly, those special cases, in which either <italic>r</italic>
<sub>fn</sub>
= 0 and <italic>r</italic>
<sub>fp</sub>
→ 1 or <italic>r</italic>
<sub>fp</sub>
= 0 and <italic>r</italic>
<sub>fn</sub>
→ 1 presented in section “<xref rid="Sec3" ref-type="sec">The MER for segmenting a single cell</xref>
”, may occur quite rarely in practice.</p>
<p>For the seven CIS algorithms employed in section “<xref rid="Sec14" ref-type="sec">The TER and its SE and CI</xref>
”, if the average MER <italic>r</italic>
<sub>a</sub>
is used, the estimated TÊRs are 0.035842, 0.037330, 0.046528, 0.058023, 0.086210, 0.087080, and 0.127707. As expected, they are correspondingly smaller than those if the weighted MER <italic>r</italic>
<sub>w</sub>
is employed (see section “<xref rid="Sec14" ref-type="sec">The TER and its SE and CI</xref>
”).</p>
<p>For the sake of discussion, if the MER is defined to be “the proportion of objects which it misclassifies” [<xref ref-type="bibr" rid="CR22">22</xref>
], <italic>r</italic>
<sub>3</sub>
= (<italic>n</italic>
<sub>g</sub>
+ <italic>n</italic>
<sub>a</sub>
) / (<italic>n</italic>
<sub>G</sub>
+ <italic>n</italic>
<sub>A</sub>
) = (<italic>n</italic>
<sub>G</sub>
x <italic>r</italic>
<sub>fn</sub>
+ <italic>n</italic>
<sub>A</sub>
x <italic>r</italic>
<sub>fp</sub>
) / (<italic>n</italic>
<sub>G</sub>
+ <italic>n</italic>
<sub>A</sub>
), which is a function of three independent variables under the constraint condition Eq. (1). Here are three observations. If <italic>n</italic>
<sub>G</sub>
= 4694, <italic>n</italic>
<sub>A</sub>
= 5276, <italic>n</italic>
<sub>g</sub>
= 16, and <italic>n</italic>
<sub>a</sub>
= 598, then <italic>r</italic>
<sub>w</sub>
= 0.110134, <italic>r</italic>
<sub>a</sub>
= 0.058376, and <italic>r</italic>
<sub>3</sub>
= 0.061585, where <italic>r</italic>
<sub>3</sub>
is very close to <italic>r</italic>
<sub>a</sub>
but almost half of <italic>r</italic>
<sub>w</sub>
. If <italic>n</italic>
<sub>G</sub>
= 1420, <italic>n</italic>
<sub>A</sub>
= 3492, <italic>n</italic>
<sub>g</sub>
= 5, and <italic>n</italic>
<sub>a</sub>
= 2077, then <italic>r</italic>
<sub>w</sub>
= 0.591308, <italic>r</italic>
<sub>a</sub>
= 0.299155, and <italic>r</italic>
<sub>3</sub>
= 0.423860, where <italic>r</italic>
<sub>3</sub>
is in the middle of <italic>r</italic>
<sub>w</sub>
and <italic>r</italic>
<sub>a</sub>
. If <italic>n</italic>
<sub>G</sub>
= 6155, <italic>n</italic>
<sub>A</sub>
= 14, <italic>n</italic>
<sub>g</sub>
= 6141, and <italic>n</italic>
<sub>a</sub>
= 0 (i.e., one of the above special cases), then <italic>r</italic>
<sub>w</sub>
= 0.997725, <italic>r</italic>
<sub>a</sub>
= 0.498863, and <italic>r</italic>
<sub>3</sub>
= 0.995461, where <italic>r</italic>
<sub>3</sub>
is close to <italic>r</italic>
<sub>w</sub>
but about twice as large as <italic>r</italic>
<sub>a</sub>
. Hence, the MER <italic>r</italic>
<sub>3</sub>
is not discussed further in this article.</p>
<p>The SE of the average MER <italic>r</italic>
<sub>a</sub>
may be estimated analytically, because the correlation coefficient between the FN rate <italic>r</italic>
<sub>fn</sub>
and the FP rate <italic>r</italic>
<sub>fp</sub>
in the CIS application is 1 as proven in the following.</p>
<p>For a GT cell and its related AD cell, assuming they are not disjoint, once the size of the FN region increases or decreases by one pixel, the size of the FP region will increase or decrease by one pixel accordingly due to the constraint condition Eq. (1). Using the notations in section “<xref rid="Sec3" ref-type="sec">The MER for segmenting a single cell</xref>
”, the correlated pairs of the FN rate r<sub>fn</sub>
and the FP rate r<sub>fp</sub>
are<disp-formula id="Equ11"><label>11</label>
<alternatives><tex-math id="M43">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \left(\ {r}_{\mathrm{fn}\ i},\ {r}_{\mathrm{fp}\ i}\right) = \left(\frac{n_{\mathrm{g}}+ i}{n_{\mathrm{G}}},\ \frac{n_{\mathrm{a}}+ i}{n_{\mathrm{A}}}\right),\ i = - m,\dots, - 1,\ 0,\ 1,\dots, n $$\end{document}</tex-math>
<mml:math id="M44"><mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfenced close=")" open="("><mml:mrow><mml:mfrac><mml:mrow><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">g</mml:mi>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">G</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
</mml:mfrac>
</mml:mrow>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
<mml:mo>,</mml:mo>
<mml:mo>…</mml:mo>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mn>0</mml:mn>
<mml:mo>,</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mn>1</mml:mn>
<mml:mo>,</mml:mo>
<mml:mo>…</mml:mo>
<mml:mo>,</mml:mo>
<mml:mi>n</mml:mi>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ11.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
where the constraints are <italic>n</italic>
<sub>g</sub>
– <italic>m</italic>
≥ 0, <italic>n</italic>
<sub>a</sub>
– <italic>m</italic>
≥ 0, <italic>n</italic>
<sub>g</sub>
+ <italic>n</italic>
≤ <italic>n</italic>
<sub>G</sub>
, <italic>n</italic>
<sub>a</sub>
+ <italic>n</italic>
≤ <italic>n</italic>
<sub>A</sub>
, and <italic>n</italic>
<sub>G</sub>
– <italic>n</italic>
<sub>g</sub>
= <italic>n</italic>
<sub>A</sub>
– <italic>n</italic>
<sub>a</sub>
.</p>
<p>The averages of the FN rate and the FP rate are,<disp-formula id="Equ12"><label>12</label>
<alternatives><tex-math id="M45">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \begin{array}{l}{\overline{r}}_{\mathrm{fn}} = \frac{1}{m+ n+1}\ {\displaystyle \sum_{i = - m}^n}\frac{n_{\mathrm{g}}+ i}{n_{\mathrm{G}}} = \frac{n_{\mathrm{g}}}{n_{\mathrm{G}}} + \frac{1}{n_{\mathrm{G}}} \times \frac{n- m}{2}\\ {}{\overline{r}}_{\mathrm{fp}} = \frac{1}{m+ n+1}\ {\displaystyle \sum_{i = - m}^n}\frac{n_{\mathrm{a}}+ i}{n_{\mathrm{A}}} = \frac{n_{\mathrm{a}}}{n_{\mathrm{A}}} + \frac{1}{n_{\mathrm{A}}} \times \frac{n- m}{2}\ .\end{array} $$\end{document}</tex-math>
<mml:math id="M46"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:msub><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">¯</mml:mo>
</mml:mover>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mn>1</mml:mn>
<mml:mrow><mml:mi>m</mml:mi>
<mml:mo>+</mml:mo>
<mml:mi>n</mml:mi>
<mml:mo>+</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mstyle displaystyle="true"><mml:munderover><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mi>n</mml:mi>
</mml:munderover>
</mml:mstyle>
<mml:mfrac><mml:mrow><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">g</mml:mi>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">G</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">g</mml:mi>
</mml:msub>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">G</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mo>+</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mn>1</mml:mn>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">G</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>×</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:mi>n</mml:mi>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:mfrac>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:msub><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">¯</mml:mo>
</mml:mover>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mn>1</mml:mn>
<mml:mrow><mml:mi>m</mml:mi>
<mml:mo>+</mml:mo>
<mml:mi>n</mml:mi>
<mml:mo>+</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mstyle displaystyle="true"><mml:munderover><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mi>n</mml:mi>
</mml:munderover>
</mml:mstyle>
<mml:mfrac><mml:mrow><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:mo>+</mml:mo>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">a</mml:mi>
</mml:msub>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mo>+</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mn>1</mml:mn>
<mml:msub><mml:mi>n</mml:mi>
<mml:mi mathvariant="normal">A</mml:mi>
</mml:msub>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>×</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:mi>n</mml:mi>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mtext>.</mml:mtext>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ12.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>Hence, the correlation coefficient is,<disp-formula id="Equ13"><label>13</label>
<alternatives><tex-math id="M47">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \begin{array}{l}\rho =\frac{{\displaystyle {\sum}_{i = - m}^n}\left(\ {r}_{\mathrm{f}\mathrm{n}\ i} - {\overline{r}}_{\mathrm{f}\mathrm{n}}\ \right)\ \left(\ {r}_{\mathrm{f}\mathrm{p}\ i} - {\overline{r}}_{\mathrm{f}\mathrm{p}}\ \right)}{\sqrt{{\displaystyle {\sum}_{i = - m}^n}{\left(\ {r}_{\mathrm{f}\mathrm{n}\ i} - {\overline{r}}_{\mathrm{f}\mathrm{n}}\ \right)}^2}\ \sqrt{{\displaystyle {\sum}_{i = - m}^n}{\left(\ {r}_{\mathrm{f} p\ i} - {\overline{r}}_{\mathrm{f} p}\ \right)}^2}}\\ {} = \frac{{\displaystyle {\sum}_{i = - m}^n}\left(\ i - \frac{n- m}{2}\ \right)\left(\ i - \frac{n- m}{2}\ \right)}{\sqrt{{\displaystyle {\sum}_{i = - m}^n}{\left(\ i - \frac{n- m}{2}\ \right)}^2}\ \sqrt{{\displaystyle {\sum}_{i = - m}^n}{\left(\ i - \frac{n- m}{2}\ \right)}^2}}=1\ .\end{array} $$\end{document}</tex-math>
<mml:math id="M48"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:mi>ρ</mml:mi>
<mml:mo>=</mml:mo>
<mml:mfrac><mml:mrow><mml:mstyle displaystyle="true"><mml:msubsup><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mrow><mml:mi>n</mml:mi>
</mml:mrow>
</mml:msubsup>
</mml:mstyle>
<mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">¯</mml:mo>
</mml:mover>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">¯</mml:mo>
</mml:mover>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">p</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow><mml:msqrt><mml:mrow><mml:mstyle displaystyle="true"><mml:msubsup><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mrow><mml:mi>n</mml:mi>
</mml:mrow>
</mml:msubsup>
</mml:mstyle>
<mml:msup><mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">¯</mml:mo>
</mml:mover>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi mathvariant="normal">n</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
<mml:mn>2</mml:mn>
</mml:msup>
</mml:mrow>
</mml:msqrt>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msqrt><mml:mrow><mml:mstyle displaystyle="true"><mml:msubsup><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mrow><mml:mi>n</mml:mi>
</mml:mrow>
</mml:msubsup>
</mml:mstyle>
<mml:msup><mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mi>r</mml:mi>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi>p</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msub><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">¯</mml:mo>
</mml:mover>
<mml:mrow><mml:mi mathvariant="normal">f</mml:mi>
<mml:mi>p</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
<mml:mn>2</mml:mn>
</mml:msup>
</mml:mrow>
</mml:msqrt>
</mml:mrow>
</mml:mfrac>
</mml:mtd>
</mml:mtr>
<mml:mtr><mml:mtd><mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:mstyle displaystyle="true"><mml:msubsup><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mrow><mml:mi>n</mml:mi>
</mml:mrow>
</mml:msubsup>
</mml:mstyle>
<mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:mi>n</mml:mi>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
<mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:mi>n</mml:mi>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
</mml:mrow>
<mml:mrow><mml:msqrt><mml:mrow><mml:mstyle displaystyle="true"><mml:msubsup><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mi>n</mml:mi>
</mml:msubsup>
</mml:mstyle>
<mml:msup><mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:mi>n</mml:mi>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
<mml:mn>2</mml:mn>
</mml:msup>
</mml:mrow>
</mml:msqrt>
<mml:mspace width="0.25em"></mml:mspace>
<mml:msqrt><mml:mrow><mml:mstyle displaystyle="true"><mml:msubsup><mml:mo stretchy="true">∑</mml:mo>
<mml:mrow><mml:mi>i</mml:mi>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>=</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mi>n</mml:mi>
</mml:msubsup>
</mml:mstyle>
<mml:msup><mml:mfenced close=")" open="("><mml:mrow><mml:mspace width="0.25em"></mml:mspace>
<mml:mi>i</mml:mi>
<mml:mo>−</mml:mo>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mfrac><mml:mrow><mml:mi>n</mml:mi>
<mml:mo>−</mml:mo>
<mml:mi>m</mml:mi>
</mml:mrow>
<mml:mn>2</mml:mn>
</mml:mfrac>
<mml:mspace width="0.25em"></mml:mspace>
</mml:mrow>
</mml:mfenced>
<mml:mn>2</mml:mn>
</mml:msup>
</mml:mrow>
</mml:msqrt>
</mml:mrow>
</mml:mfrac>
<mml:mo>=</mml:mo>
<mml:mn>1</mml:mn>
<mml:mspace width="0.25em"></mml:mspace>
<mml:mtext>.</mml:mtext>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
<graphic xlink:href="12859_2017_1527_Article_Equ13.gif" position="anchor"></graphic>
</alternatives>
</disp-formula>
</p>
<p>Further, using the first formula of Eq. (3), the SE of the average MER <italic>r</italic>
<sub>a</sub>
turns out to be SÊ<sub>a</sub>
= (SÊ<sub>fn</sub>
+ SÊ<sub>fp</sub>
) / 2, in which SÊ<sub>fn</sub>
and SÊ<sub>fp</sub>
may be estimated using SÊ = sqrt [<inline-formula id="IEq12"><alternatives><tex-math id="M49">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{r} $$\end{document}</tex-math>
<mml:math id="M50"><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq12.gif"></inline-graphic>
</alternatives>
</inline-formula>
(1 – <inline-formula id="IEq13"><alternatives><tex-math id="M51">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{r} $$\end{document}</tex-math>
<mml:math id="M52"><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq13.gif"></inline-graphic>
</alternatives>
</inline-formula>
) / <italic>n</italic>
], where <inline-formula id="IEq14"><alternatives><tex-math id="M53">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{r} $$\end{document}</tex-math>
<mml:math id="M54"><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq14.gif"></inline-graphic>
</alternatives>
</inline-formula>
= <italic>r</italic>
<sub>fn</sub>
and <italic>n</italic>
= <italic>n</italic>
<sub>G</sub>
for SÊ<sub>fn</sub>
, and <inline-formula id="IEq15"><alternatives><tex-math id="M55">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ \widehat{r} $$\end{document}</tex-math>
<mml:math id="M56"><mml:mover accent="true"><mml:mi>r</mml:mi>
<mml:mo stretchy="true">^</mml:mo>
</mml:mover>
</mml:math>
<inline-graphic xlink:href="12859_2017_1527_Article_IEq15.gif"></inline-graphic>
</alternatives>
</inline-formula>
= <italic>r</italic>
<sub>fp</sub>
and <italic>n</italic>
= <italic>n</italic>
<sub>A</sub>
for SÊ<sub>fp</sub>
.</p>
<p>However, such an analytical approach generally underestimates the SE of MER, and thus the SE of TER (see Eq. (7)) as opposed to the bootstrap method. For the seven CIS algorithms, if the bootstrap method is employed, the estimated SÊs of TER, in which the average MER <italic>r</italic>
<sub>a</sub>
is used, are 0.001001, 0.000246, 0.000537, 0.000254, 0.001888, 0.000820, and 0.000292, respectively. If the analytical approach is used, they are 0.000169, 0.000181, 0.000180, 0.000188, 0.000196, 0.000219, and 0.000208, respectively.</p>
</sec>
<sec id="Sec18" sec-type="conclusion"><title>Conclusions</title>
<p>Our novel performance measure TER comes with SE and 95% CI without restrictions on data size, and the correlation coefficient of TERs between two CIS algorithms is also solved. Thus, the well-established statistical approach can be carried out to evaluate and compare the performance levels of CIS algorithms with statistical confidence. Significance values for differences in CIS algorithm performance in combination with other factors such as computational execution time, etc. can be used as a basis for selecting algorithms.</p>
<p>No matter which MER is chosen and no matter which CIS algorithms and datasets are employed, the approaches of designing the TER using the total probability in statistics based on MER, computing SE and 95% CI of TER based on using the bootstrap method to estimate the SE of MER, and conducting hypothesis testing, etc. explored in this article remain intact. The TER <italic>ε</italic>
aggregates all MERs weighted by the size of a cell divided by the total sizes of all cells so that the algorithm pays more penalties if making errors while segmenting larger cells.</p>
<p>The TER <italic>ε</italic>
is supported by the pairwise-comparison test of MERs using 106 manually segmented GT cells with different sizes and seven CIS algorithms taken from ImageJ. It is also qualitatively consistent with the observations from the MER histograms. The TER approach is more effective than the bivariate approach and the CDF approach.</p>
<p>The SE and 95% CI of the TER are computed using Eq. (7), based on the SE of MER that is calculated using the bootstrap method under a constraint condition for CIS during bootstrap resampling. The nature of the bootstrap method is stochastic. However, our studies reveal that the variation of the estimated SÊ of TER is small. Moreover, in our studies, all estimated SÊs of TERs obtained by a random execution of bootstrap method while computing the SÊs of MERs fall in the 95% CÎ of the estimated SÊs of TERs correspondingly.</p>
<p>The error bars of the TERs can be used to evaluate the performance level of a CIS algorithm against a hypothesized value, and classify CIS algorithms into different classes in terms of performance accuracies based on the criteria of performance accuracies. While the error bars overlap, the two-algorithm hypothesis testing can be employed to compare two CIS algorithms and determine the statistical significance of their performance difference. The Z test in Eq. (9) involves not only the SEs of TERs but also the correlation coefficient between the TERs of two CIS algorithms, which are all investigated in this article.</p>
</sec>
</body>
<back><glossary><title>Abbreviations</title>
<def-list><def-item><term>AD</term>
<def><p>Algorithm detected</p>
</def>
</def-item>
<def-item><term>CDF</term>
<def><p>Cumulative distribution function</p>
</def>
</def-item>
<def-item><term>CI</term>
<def><p>Confidence interval</p>
</def>
</def-item>
<def-item><term>CIS</term>
<def><p>Cell image segmentation</p>
</def>
</def-item>
<def-item><term>FN</term>
<def><p>False negative</p>
</def>
</def-item>
<def-item><term>FP</term>
<def><p>False positive</p>
</def>
</def-item>
<def-item><term>GT</term>
<def><p>Ground truth</p>
</def>
</def-item>
<def-item><term>MER</term>
<def><p>Misclassification error rates</p>
</def>
</def-item>
<def-item><term>ROC</term>
<def><p>Receiver operating characteristic</p>
</def>
</def-item>
<def-item><term>SE</term>
<def><p>Standard error</p>
</def>
</def-item>
<def-item><term>TER</term>
<def><p>Total error rate</p>
</def>
</def-item>
<def-item><term>WR</term>
<def><p>With replacement</p>
</def>
</def-item>
</def-list>
</glossary>
<fn-group><fn id="Fn1"><label>1</label>
<p>Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.</p>
</fn>
</fn-group>
<ack><title>Acknowledgment</title>
<p>Not applicable.</p>
<sec id="FPar1"><title>Funding</title>
<p>This work was funded entirely by the National Institute of Standards and Technology.</p>
</sec>
<sec id="FPar2"><title>Availability of data and materials</title>
<p>The software and test data is available at <ext-link ext-link-type="uri" xlink:href="https://github.com/usnistgov/Cell-Imaging">https://github.com/usnistgov/Cell-Imaging</ext-link>
.</p>
</sec>
<sec id="FPar3"><title>Authors’ contributions</title>
<p>JCW contributed to conception, designing algorithms, developing formulas, computer programming, and manuscript preparation. MH contributed to acquiring the data, writing an ImageJ macro code for computing GT areas, AD areas, FPs and FNs in fluorescent microscopy images, and manuscript preparation. RNK contributed to initiation of the project, and manuscript preparation. JTE contributed to acquiring the data, and manuscript preparation. ALP contributed to initiation of the project, and acquiring the data. All authors have read and approved the final version of this manuscript.</p>
</sec>
<sec id="FPar4"><title>Competing interests</title>
<p>The authors declare that they have no competing interests.</p>
</sec>
<sec id="FPar5"><title>Consent for publication</title>
<p>Not applicable.</p>
</sec>
<sec id="FPar6"><title>Ethics approval and consent to participate</title>
<p>Not applicable.</p>
</sec>
</ack>
<ref-list id="Bib1"><title>References</title>
<ref id="CR1"><label>1.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Hill</surname>
<given-names>AA</given-names>
</name>
<name><surname>LaPan</surname>
<given-names>P</given-names>
</name>
<name><surname>Li</surname>
<given-names>Y</given-names>
</name>
<name><surname>Haney</surname>
<given-names>S</given-names>
</name>
</person-group>
<article-title>Impact of image segmentation on high-content screening data quality for SK-BR-3 cells</article-title>
<source>BMC Bioinf.</source>
<year>2007</year>
<volume>8</volume>
<fpage>340</fpage>
<pub-id pub-id-type="doi">10.1186/1471-2105-8-340</pub-id>
</element-citation>
</ref>
<ref id="CR2"><label>2.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Chalfoun</surname>
<given-names>J</given-names>
</name>
<name><surname>Kociolek</surname>
<given-names>M</given-names>
</name>
<name><surname>Dima</surname>
<given-names>A</given-names>
</name>
<name><surname>Cardone</surname>
<given-names>A</given-names>
</name>
<name><surname>Peskin</surname>
<given-names>A</given-names>
</name>
<name><surname>Bajcsy</surname>
<given-names>P</given-names>
</name>
<name><surname>Brady</surname>
<given-names>M</given-names>
</name>
</person-group>
<article-title>Segmenting time-lapse phase contrast images of adjacent NIH 3 T3 cells</article-title>
<source>J Microsc</source>
<year>2013</year>
<volume>249</volume>
<issue>1</issue>
<fpage>41</fpage>
<lpage>52</lpage>
<pub-id pub-id-type="doi">10.1111/j.1365-2818.2012.03678.x</pub-id>
<pub-id pub-id-type="pmid">23126432</pub-id>
</element-citation>
</ref>
<ref id="CR3"><label>3.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bajcsy</surname>
<given-names>P</given-names>
</name>
<name><surname>Cardone</surname>
<given-names>A</given-names>
</name>
<name><surname>Chalfoun</surname>
<given-names>J</given-names>
</name>
<name><surname>Halter</surname>
<given-names>M</given-names>
</name>
<name><surname>Juba</surname>
<given-names>D</given-names>
</name>
<name><surname>Kociolek</surname>
<given-names>M</given-names>
</name>
<name><surname>Majurski</surname>
<given-names>M</given-names>
</name>
<name><surname>Peskin</surname>
<given-names>A</given-names>
</name>
<name><surname>Simon</surname>
<given-names>CG</given-names>
<suffix>Jr</suffix>
</name>
<name><surname>Simon</surname>
<given-names>M</given-names>
</name>
<name><surname>Vandecreme</surname>
<given-names>A</given-names>
</name>
<name><surname>Brady</surname>
<given-names>M</given-names>
</name>
</person-group>
<article-title>Survey statistics of automated segmentations applied to optical imaging of mammalian cells</article-title>
<source>BMC Bioinf.</source>
<year>2015</year>
<volume>16</volume>
<fpage>330</fpage>
<pub-id pub-id-type="doi">10.1186/s12859-015-0762-2</pub-id>
</element-citation>
</ref>
<ref id="CR4"><label>4.</label>
<element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Ostle</surname>
<given-names>B</given-names>
</name>
<name><surname>Malone</surname>
<given-names>LC</given-names>
</name>
</person-group>
<source>Statistics in research: basic concepts and techniques for research workers</source>
<year>1988</year>
<edition>4</edition>
<publisher-loc>Ames</publisher-loc>
<publisher-name>Iowa State University Press</publisher-name>
</element-citation>
</ref>
<ref id="CR5"><label>5.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rand</surname>
<given-names>WM</given-names>
</name>
</person-group>
<article-title>Objective criteria for the evaluation of clustering methods</article-title>
<source>J Am Stat Assoc</source>
<year>1971</year>
<volume>66</volume>
<fpage>846</fpage>
<lpage>50</lpage>
<pub-id pub-id-type="doi">10.1080/01621459.1971.10482356</pub-id>
</element-citation>
</ref>
<ref id="CR6"><label>6.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Unnikrishnan</surname>
<given-names>R</given-names>
</name>
<name><surname>Pantofaru</surname>
<given-names>C</given-names>
</name>
<name><surname>Hebert</surname>
<given-names>M</given-names>
</name>
</person-group>
<article-title>Toward objective evaluation of image segmentation algorithms</article-title>
<source>IEEE Trans Pattern Anal Mach Intell</source>
<year>2007</year>
<volume>29</volume>
<fpage>929</fpage>
<lpage>44</lpage>
<pub-id pub-id-type="doi">10.1109/TPAMI.2007.1046</pub-id>
<pub-id pub-id-type="pmid">17431294</pub-id>
</element-citation>
</ref>
<ref id="CR7"><label>7.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Zijdenbos</surname>
<given-names>AP</given-names>
</name>
<name><surname>Dawant</surname>
<given-names>BM</given-names>
</name>
<name><surname>Margolin</surname>
<given-names>RA</given-names>
</name>
<name><surname>Palmer</surname>
<given-names>AC</given-names>
</name>
</person-group>
<article-title>Morphometric analysis of white matter lesions in MR images: method and validation</article-title>
<source>IEEE Trans Med Imag</source>
<year>1994</year>
<volume>13</volume>
<fpage>716</fpage>
<lpage>24</lpage>
<pub-id pub-id-type="doi">10.1109/42.363096</pub-id>
</element-citation>
</ref>
<ref id="CR8"><label>8.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Zhang</surname>
<given-names>YJ</given-names>
</name>
</person-group>
<article-title>Evaluation and comparison of different segmentation algorithms</article-title>
<source>Pattern Recognit Lett</source>
<year>1997</year>
<volume>18</volume>
<fpage>963</fpage>
<lpage>74</lpage>
<pub-id pub-id-type="doi">10.1016/S0167-8655(97)00083-4</pub-id>
</element-citation>
</ref>
<ref id="CR9"><label>9.</label>
<mixed-citation publication-type="other">Coelho LP, Shariff A, Murphy RF. Nuclear segmentation in microscope cell images: A hand segmented dataset and comparison of algorithms. Proc IEEE Int Symp Biomed Imag. 2009;518–521.</mixed-citation>
</ref>
<ref id="CR10"><label>10.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Klar</surname>
<given-names>N</given-names>
</name>
<name><surname>Lipsitz</surname>
<given-names>SR</given-names>
</name>
<name><surname>Parzen</surname>
<given-names>M</given-names>
</name>
<name><surname>Leong</surname>
<given-names>T</given-names>
</name>
</person-group>
<article-title>An exact bootstrap confidence interval for κ in small samples</article-title>
<source>Statistician</source>
<year>2002</year>
<volume>51</volume>
<fpage>467</fpage>
<lpage>78</lpage>
</element-citation>
</ref>
<ref id="CR11"><label>11.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dimopoulos</surname>
<given-names>S</given-names>
</name>
<name><surname>Mayer</surname>
<given-names>CE</given-names>
</name>
<name><surname>Rudolf</surname>
<given-names>F</given-names>
</name>
<name><surname>Stelling</surname>
<given-names>J</given-names>
</name>
</person-group>
<article-title>Accurate cell segmentation in microscopy images using membrane patterns</article-title>
<source>Bioinformatics</source>
<year>2014</year>
<volume>30</volume>
<fpage>2644</fpage>
<lpage>51</lpage>
<pub-id pub-id-type="doi">10.1093/bioinformatics/btu302</pub-id>
<pub-id pub-id-type="pmid">24849580</pub-id>
</element-citation>
</ref>
<ref id="CR12"><label>12.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Zhang</surname>
<given-names>H</given-names>
</name>
<name><surname>Fritts</surname>
<given-names>JE</given-names>
</name>
<name><surname>Goldman</surname>
<given-names>SA</given-names>
</name>
</person-group>
<article-title>Image segmentation evaluation: a survey of unsupervised methods</article-title>
<source>Comput. Vis. Image Underst.</source>
<year>2008</year>
<volume>110</volume>
<fpage>260</fpage>
<lpage>80</lpage>
<pub-id pub-id-type="doi">10.1016/j.cviu.2007.08.003</pub-id>
</element-citation>
</ref>
<ref id="CR13"><label>13.</label>
<mixed-citation publication-type="other">Rasband WS. ImageJ. National Institutes of Health, Bethesda, Maryland, USA. <ext-link ext-link-type="uri" xlink:href="https://imagej.net/">https://imagej.net/</ext-link>
. Accessed 14 Feb 2017.</mixed-citation>
</ref>
<ref id="CR14"><label>14.</label>
<mixed-citation publication-type="other">National Institute of Standards and Technology, Gaithersburg, Maryland, USA. <ext-link ext-link-type="uri" xlink:href="https://randr.nist.gov/cellimage/">https://randr.nist.gov/cellimage/</ext-link>
. Accessed 14 Feb 2017. </mixed-citation>
</ref>
<ref id="CR15"><label>15.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Efron</surname>
<given-names>B</given-names>
</name>
</person-group>
<article-title>Bootstrap methods: another look at the Jackknife</article-title>
<source>Ann Statistics</source>
<year>1979</year>
<volume>7</volume>
<fpage>1</fpage>
<lpage>26</lpage>
<pub-id pub-id-type="doi">10.1214/aos/1176344552</pub-id>
</element-citation>
</ref>
<ref id="CR16"><label>16.</label>
<element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Efron</surname>
<given-names>B</given-names>
</name>
<name><surname>Tibshirani</surname>
<given-names>RJ</given-names>
</name>
</person-group>
<source>An Introduction to the Bootstrap</source>
<year>1993</year>
<publisher-loc>New York</publisher-loc>
<publisher-name>Chapman & Hall</publisher-name>
</element-citation>
</ref>
<ref id="CR17"><label>17.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Wu</surname>
<given-names>JC</given-names>
</name>
<name><surname>Martin</surname>
<given-names>AF</given-names>
</name>
<name><surname>Kacker</surname>
<given-names>RN</given-names>
</name>
</person-group>
<article-title>Measures, uncertainties, and significance test in operational ROC analysis</article-title>
<source>J Res Natl Inst Stand Technol</source>
<year>2011</year>
<volume>116</volume>
<fpage>517</fpage>
<lpage>37</lpage>
<pub-id pub-id-type="doi">10.6028/jres.116.003</pub-id>
<pub-id pub-id-type="pmid">26989582</pub-id>
</element-citation>
</ref>
<ref id="CR18"><label>18.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Wu</surname>
<given-names>JC</given-names>
</name>
<name><surname>Martin</surname>
<given-names>AF</given-names>
</name>
<name><surname>Kacker</surname>
<given-names>RN</given-names>
</name>
</person-group>
<article-title>Bootstrap variability studies in ROC analysis on large datasets</article-title>
<source>Commun. Stat. Simul. Comput.</source>
<year>2014</year>
<volume>43</volume>
<issue>1</issue>
<fpage>225</fpage>
<lpage>36</lpage>
<pub-id pub-id-type="doi">10.1080/03610918.2012.700362</pub-id>
</element-citation>
</ref>
<ref id="CR19"><label>19.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Wu</surname>
<given-names>JC</given-names>
</name>
<name><surname>Martin</surname>
<given-names>AF</given-names>
</name>
<name><surname>Kacker</surname>
<given-names>RN</given-names>
</name>
</person-group>
<article-title>Validation of nonparametric two-sample bootstrap in ROC analysis on large datasets</article-title>
<source>Commun. Stat. Simul. Comput.</source>
<year>2016</year>
<volume>45</volume>
<issue>5</issue>
<fpage>1689</fpage>
<lpage>703</lpage>
<pub-id pub-id-type="doi">10.1080/03610918.2015.1065327</pub-id>
<pub-id pub-id-type="pmid">27499571</pub-id>
</element-citation>
</ref>
<ref id="CR20"><label>20.</label>
<mixed-citation publication-type="other">Wu JC, Martin AF, Greenberg CS, Kacker RN. The impact of data dependence on speaker recognition evaluation. IEEE/ACM Trans. Audio, Speech, Lang. Process. 2017;25(1):5-18.</mixed-citation>
</ref>
<ref id="CR21"><label>21.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dresen</surname>
<given-names>IMG</given-names>
</name>
<name><surname>Boes</surname>
<given-names>T</given-names>
</name>
<name><surname>Huesing</surname>
<given-names>J</given-names>
</name>
<name><surname>Neuhaeuser</surname>
<given-names>M</given-names>
</name>
<name><surname>Joeckel</surname>
<given-names>KH</given-names>
</name>
</person-group>
<article-title>New resampling method for evaluating stability of clusters</article-title>
<source>BMC Bioinf.</source>
<year>2008</year>
<volume>9</volume>
<fpage>42</fpage>
<pub-id pub-id-type="doi">10.1186/1471-2105-9-42</pub-id>
</element-citation>
</ref>
<ref id="CR22"><label>22.</label>
<element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Hand</surname>
<given-names>DJ</given-names>
</name>
</person-group>
<source>Construction and assessment of classification rules</source>
<year>1997</year>
<publisher-loc>New York</publisher-loc>
<publisher-name>Wiley</publisher-name>
</element-citation>
</ref>
<ref id="CR23"><label>23.</label>
<mixed-citation publication-type="other"><ext-link ext-link-type="uri" xlink:href="http://mathworld.wolfram.com/Mean.html">http://mathworld.wolfram.com/Mean.html</ext-link>
, https://en.wikipedia.org/wiki/Inequality_of_arithmetic_and_geometric_means, https://en.wikipedia.org/wiki/Harmonic_mean. Accessed 17 Jan 2017.</mixed-citation>
</ref>
<ref id="CR24"><label>24.</label>
<element-citation publication-type="book"><person-group person-group-type="author"><name><surname>van der Waerden</surname>
<given-names>BL</given-names>
</name>
</person-group>
<source>Mathematical statistics</source>
<year>1969</year>
<publisher-loc>Berlin</publisher-loc>
<publisher-name>Springer</publisher-name>
</element-citation>
</ref>
<ref id="CR25"><label>25.</label>
<mixed-citation publication-type="other">Zwillinger D, Kokoska S. CRC Standard Probability and Statistics Tables and Formulae. New York: Chapman & Hall/CRC Press; 2000.</mixed-citation>
</ref>
<ref id="CR26"><label>26.</label>
<element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Rice</surname>
<given-names>JA</given-names>
</name>
</person-group>
<source>Mathematical statistics and data analysis</source>
<year>2006</year>
<edition>3</edition>
<publisher-loc>Belmont</publisher-loc>
<publisher-name>Duxbury Advanced</publisher-name>
</element-citation>
</ref>
<ref id="CR27"><label>27.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Hyndman</surname>
<given-names>RJ</given-names>
</name>
<name><surname>Fan</surname>
<given-names>Y</given-names>
</name>
</person-group>
<article-title>Sample quantiles in statistical packages</article-title>
<source>Am Stat</source>
<year>1996</year>
<volume>50</volume>
<fpage>361</fpage>
<lpage>5</lpage>
</element-citation>
</ref>
</ref-list>
</back>
</pmc>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Sante/explor/MersV1/Data/Pmc/Corpus
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000265 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd -nk 000265 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Sante |area= MersV1 |flux= Pmc |étape= Corpus |type= RBID |clé= PMC:5351215 |texte= A novel measure and significance testing in data analysis of cell image segmentation }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/RBID.i -Sk "pubmed:28292256" \ | HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd \ | NlmPubMed2Wicri -a MersV1
![]() | This area was generated with Dilib version V0.6.33. | ![]() |