Links to Exploration step
Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Evaluating a linear <bold><italic>k</italic></bold>-mer model for protein-DNA interactions using high-throughput SELEX data</title><author><name sortKey="K H R, Juhani" sort="K H R, Juhani" uniqKey="K H R J" first="Juhani" last="K H R">Juhani K H R</name><affiliation><nlm:aff id="I1">Department of Information and Computer Science, Aalto University School of Science, FI-00076 Aalto, Finland</nlm:aff></affiliation></author><author><name sortKey="L Hdesm Ki, Harri" sort="L Hdesm Ki, Harri" uniqKey="L Hdesm Ki H" first="Harri" last="L Hdesm Ki">Harri L Hdesm Ki</name><affiliation><nlm:aff id="I1">Department of Information and Computer Science, Aalto University School of Science, FI-00076 Aalto, Finland</nlm:aff></affiliation><affiliation><nlm:aff id="I2">Turku Centre for Biotechnology, Turku University, Turku, Finland</nlm:aff></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">PMC</idno><idno type="pmid">24267147</idno><idno type="pmc">3750486</idno><idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3750486</idno><idno type="RBID">PMC:3750486</idno><idno type="doi">10.1186/1471-2105-14-S10-S2</idno><date when="2013">2013</date><idno type="wicri:Area/Pmc/Corpus">000948</idno><idno type="wicri:explorRef" wicri:stream="Pmc" wicri:step="Corpus" wicri:corpus="PMC">000948</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Evaluating a linear <bold><italic>k</italic></bold>-mer model for protein-DNA interactions using high-throughput SELEX data</title><author><name sortKey="K H R, Juhani" sort="K H R, Juhani" uniqKey="K H R J" first="Juhani" last="K H R">Juhani K H R</name><affiliation><nlm:aff id="I1">Department of Information and Computer Science, Aalto University School of Science, FI-00076 Aalto, Finland</nlm:aff></affiliation></author><author><name sortKey="L Hdesm Ki, Harri" sort="L Hdesm Ki, Harri" uniqKey="L Hdesm Ki H" first="Harri" last="L Hdesm Ki">Harri L Hdesm Ki</name><affiliation><nlm:aff id="I1">Department of Information and Computer Science, Aalto University School of Science, FI-00076 Aalto, Finland</nlm:aff></affiliation><affiliation><nlm:aff id="I2">Turku Centre for Biotechnology, Turku University, Turku, Finland</nlm:aff></affiliation></author></analytic><series><title level="j">BMC Bioinformatics</title><idno type="eISSN">1471-2105</idno><imprint><date when="2013">2013</date></imprint></series></biblStruct></sourceDesc></fileDesc><profileDesc><textClass></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en"><p>Transcription factor (TF) binding to DNA can be modeled in a number of different ways. It is highly debated which modeling methods are the best, how the models should be built and what can they be applied to. In this study a linear <italic>k</italic>-mer model proposed for predicting TF specificity in protein binding microarrays (PBM) is applied to a high-throughput SELEX data and the question of how to choose the most informative <italic>k</italic>-mers to the binding model is studied. We implemented the standard cross-validation scheme to reduce the number of <italic>k</italic>-mers in the model and observed that the number of <italic>k</italic>-mers can often be reduced significantly without a great negative effect on prediction accuracy. We also found that the later SELEX enrichment cycles provide a much better discrimination between bound and unbound sequences as model prediction accuracies increased for all proteins together with the cycle number. We compared prediction performance of <italic>k</italic>-mer and position specific weight matrix (PWM) models derived from the same SELEX data. Consistent with previous results on PBM data, performance of the <italic>k</italic>-mer model was on average 9%-units better. For the 15 proteins in the SELEX data set with medium enrichment cycles, classification accuracies were on average 71% and 62% for <italic>k</italic>-mer and PWMs, respectively. Finally, the <italic>k</italic>-mer model trained with SELEX data was evaluated on ChIP-seq data demonstrating substantial improvements for some proteins. For protein GATA1 the model can distinquish between true ChIP-seq peaks and negative peaks. For proteins RFX3 and NFATC1 the performance of the model was no better than chance.</p></div></front><back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Wasserman, W" uniqKey="Wasserman W">W Wasserman</name></author><author><name sortKey="Sandelin, A" uniqKey="Sandelin A">A Sandelin</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Weirauch, M" uniqKey="Weirauch M">M Weirauch</name></author><author><name sortKey="Cote, A" uniqKey="Cote A">A Cote</name></author><author><name sortKey="Norel, R" uniqKey="Norel R">R Norel</name></author><author><name sortKey="Annala, M" uniqKey="Annala M">M Annala</name></author><author><name sortKey="Zhao, Y" uniqKey="Zhao Y">Y Zhao</name></author><author><name sortKey="Riley, T" uniqKey="Riley T">T Riley</name></author><author><name sortKey="Saez Rodriguez, J" uniqKey="Saez Rodriguez J">J Saez-Rodriguez</name></author><author><name sortKey="Cokelaer, T" uniqKey="Cokelaer T">T Cokelaer</name></author><author><name sortKey="Vedenko, A" uniqKey="Vedenko A">A Vedenko</name></author><author><name sortKey="Talukder, S" uniqKey="Talukder S">S Talukder</name></author><author><name sortKey="Bussemaker, H" uniqKey="Bussemaker H">H Bussemaker</name></author><author><name sortKey="Morris, Q" uniqKey="Morris Q">Q Morris</name></author><author><name sortKey="Bulyk, M" uniqKey="Bulyk M">M Bulyk</name></author><author><name sortKey="Stolovitzky, G" uniqKey="Stolovitzky G">G Stolovitzky</name></author><author><name sortKey="Hughes, T" uniqKey="Hughes T">T Hughes</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Staden, R" uniqKey="Staden R">R Staden</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Stormo, G" uniqKey="Stormo G">G Stormo</name></author><author><name sortKey="Schneider, T" uniqKey="Schneider T">T Schneider</name></author><author><name sortKey="Gold, L" uniqKey="Gold L">L Gold</name></author><author><name sortKey="Ehrenfeucht, A" uniqKey="Ehrenfeucht A">A Ehrenfeucht</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bailey, T" uniqKey="Bailey T">T Bailey</name></author><author><name sortKey="Elkan, C" uniqKey="Elkan C">C Elkan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Lawrence, C" uniqKey="Lawrence C">C Lawrence</name></author><author><name sortKey="Altschul, S" uniqKey="Altschul S">S Altschul</name></author><author><name sortKey="Boguski, M" uniqKey="Boguski M">M Boguski</name></author><author><name sortKey="Liu, J" uniqKey="Liu J">J Liu</name></author><author><name sortKey="Neuwald, A" uniqKey="Neuwald A">A Neuwald</name></author><author><name sortKey="Wootton, J" uniqKey="Wootton J">J Wootton</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wingender, E" uniqKey="Wingender E">E Wingender</name></author><author><name sortKey="Chen, X" uniqKey="Chen X">X Chen</name></author><author><name sortKey="Hehl, R" uniqKey="Hehl R">R Hehl</name></author><author><name sortKey="Karas, H" uniqKey="Karas H">H Karas</name></author><author><name sortKey="Liebich, I" uniqKey="Liebich I">I Liebich</name></author><author><name sortKey="Matys, V" uniqKey="Matys V">V Matys</name></author><author><name sortKey="Meinhardt, T" uniqKey="Meinhardt T">T Meinhardt</name></author><author><name sortKey="Pruss, M" uniqKey="Pruss M">M Prüss</name></author><author><name sortKey="Reuter, I" uniqKey="Reuter I">I Reuter</name></author><author><name sortKey="Schacherer, F" uniqKey="Schacherer F">F Schacherer</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Sandelin, A" uniqKey="Sandelin A">A Sandelin</name></author><author><name sortKey="Alkema, W" uniqKey="Alkema W">W Alkema</name></author><author><name sortKey="Engstrom, P" uniqKey="Engstrom P">P Engström</name></author><author><name sortKey="Wasserman, W" uniqKey="Wasserman W">W Wasserman</name></author><author><name sortKey="Lenhard, B" uniqKey="Lenhard B">B Lenhard</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Annala, M" uniqKey="Annala M">M Annala</name></author><author><name sortKey="Laurila, K" uniqKey="Laurila K">K Laurila</name></author><author><name sortKey="L Hdesm Ki, H" uniqKey="L Hdesm Ki H">H Lähdesmäki</name></author><author><name sortKey="Nykter, M" uniqKey="Nykter M">M Nykter</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Jolma, A" uniqKey="Jolma A">A Jolma</name></author><author><name sortKey="Kivioja, T" uniqKey="Kivioja T">T Kivioja</name></author><author><name sortKey="Toivonen, J" uniqKey="Toivonen J">J Toivonen</name></author><author><name sortKey="Cheng, L" uniqKey="Cheng L">L Cheng</name></author><author><name sortKey="Wei, G" uniqKey="Wei G">G Wei</name></author><author><name sortKey="Enge, M" uniqKey="Enge M">M Enge</name></author><author><name sortKey="Taipale, M" uniqKey="Taipale M">M Taipale</name></author><author><name sortKey="Vaquerizas, J" uniqKey="Vaquerizas J">J Vaquerizas</name></author><author><name sortKey="Yan, J" uniqKey="Yan J">J Yan</name></author><author><name sortKey="Sillanp, M" uniqKey="Sillanp M">M Sillanpää</name></author><author><name sortKey="Bonke, M" uniqKey="Bonke M">M Bonke</name></author><author><name sortKey="Palin, K" uniqKey="Palin K">K Palin</name></author><author><name sortKey="Talukder, S" uniqKey="Talukder S">S Talukder</name></author><author><name sortKey="Hughes, T" uniqKey="Hughes T">T Hughes</name></author><author><name sortKey="Luscombe, N" uniqKey="Luscombe N">N Luscombe</name></author><author><name sortKey="Ukkonen, E" uniqKey="Ukkonen E">E Ukkonen</name></author><author><name sortKey="Taipale, J" uniqKey="Taipale J">J Taipale</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Ambroise, C" uniqKey="Ambroise C">C Ambroise</name></author><author><name sortKey="Mclachlan, G" uniqKey="Mclachlan G">G McLachlan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Gengsheng, Q" uniqKey="Gengsheng Q">Q Gengsheng</name></author><author><name sortKey="Hotilovac, L" uniqKey="Hotilovac L">L Hotilovac</name></author></analytic></biblStruct></listBibl></div1></back></TEI><pmc article-type="research-article" xml:lang="en"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">BMC Bioinformatics</journal-id><journal-id journal-id-type="iso-abbrev">BMC Bioinformatics</journal-id><journal-title-group><journal-title>BMC Bioinformatics</journal-title></journal-title-group><issn pub-type="epub">1471-2105</issn><publisher><publisher-name>BioMed Central</publisher-name></publisher></journal-meta><article-meta><article-id pub-id-type="pmid">24267147</article-id><article-id pub-id-type="pmc">3750486</article-id><article-id pub-id-type="publisher-id">1471-2105-14-S10-S2</article-id><article-id pub-id-type="doi">10.1186/1471-2105-14-S10-S2</article-id><article-categories><subj-group subj-group-type="heading"><subject>Research</subject></subj-group></article-categories><title-group><article-title>Evaluating a linear <bold><italic>k</italic></bold>-mer model for protein-DNA interactions using high-throughput SELEX data</article-title></title-group><contrib-group><contrib contrib-type="author" corresp="yes" id="A1"><name><surname>Kähärä</surname><given-names>Juhani</given-names></name><xref ref-type="aff" rid="I1">1</xref><email>juhani.kahara@aalto.fi</email></contrib><contrib contrib-type="author" id="A2"><name><surname>Lähdesmäki</surname><given-names>Harri</given-names></name><xref ref-type="aff" rid="I1">1</xref><xref ref-type="aff" rid="I2">2</xref><email>harri.lahdesmaki@aalto.fi</email></contrib></contrib-group><aff id="I1"><label>1</label>Department of Information and Computer Science, Aalto University School of Science, FI-00076 Aalto, Finland</aff><aff id="I2"><label>2</label>Turku Centre for Biotechnology, Turku University, Turku, Finland</aff><pub-date pub-type="collection"><year>2013</year></pub-date><pub-date pub-type="epub"><day>12</day><month>8</month><year>2013</year></pub-date><volume>14</volume><issue>Suppl 10</issue><supplement><named-content content-type="supplement-title">Selected articles from the 10th International Workshop on Computational Systems Biology (WCSB) 2013: Bioinformatics</named-content><named-content content-type="supplement-editor">Reija Autio, Ilya Shmulevich, Korbinian Strimmer and Carsten Wiuf</named-content><named-content content-type="supplement-sponsor">Publication of this supplement has not been supported by sponsorship. Information about the source of funding for publication charges can be found in the individual articles. Articles have undergone the journal's standard peer-review process for supplements. The Supplement Editors declare that they have no competing interests.</named-content></supplement><fpage>S2</fpage><lpage>S2</lpage><permissions><copyright-statement>Copyright © 2013 Kähärä and Lähdesmäki; licensee BioMed Central Ltd.</copyright-statement><copyright-year>2013</copyright-year><copyright-holder>Kähärä and Lähdesmäki; licensee BioMed Central Ltd.</copyright-holder><license license-type="open-access" xlink:href="http://creativecommons.org/licenses/by/2.0"><license-p>This is an Open Access article distributed under the terms of the Creative Commons Attribution License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/2.0">http://creativecommons.org/licenses/by/2.0</ext-link>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p></license></permissions><self-uri xlink:href="http://www.biomedcentral.com/1471-2105/14/S10/S2"></self-uri><abstract><p>Transcription factor (TF) binding to DNA can be modeled in a number of different ways. It is highly debated which modeling methods are the best, how the models should be built and what can they be applied to. In this study a linear <italic>k</italic>-mer model proposed for predicting TF specificity in protein binding microarrays (PBM) is applied to a high-throughput SELEX data and the question of how to choose the most informative <italic>k</italic>-mers to the binding model is studied. We implemented the standard cross-validation scheme to reduce the number of <italic>k</italic>-mers in the model and observed that the number of <italic>k</italic>-mers can often be reduced significantly without a great negative effect on prediction accuracy. We also found that the later SELEX enrichment cycles provide a much better discrimination between bound and unbound sequences as model prediction accuracies increased for all proteins together with the cycle number. We compared prediction performance of <italic>k</italic>-mer and position specific weight matrix (PWM) models derived from the same SELEX data. Consistent with previous results on PBM data, performance of the <italic>k</italic>-mer model was on average 9%-units better. For the 15 proteins in the SELEX data set with medium enrichment cycles, classification accuracies were on average 71% and 62% for <italic>k</italic>-mer and PWMs, respectively. Finally, the <italic>k</italic>-mer model trained with SELEX data was evaluated on ChIP-seq data demonstrating substantial improvements for some proteins. For protein GATA1 the model can distinquish between true ChIP-seq peaks and negative peaks. For proteins RFX3 and NFATC1 the performance of the model was no better than chance.</p></abstract><conference><conf-date>10-12 June 2013</conf-date><conf-name>10th International Workshop on Computational Systems Biology</conf-name><conf-loc>Tampere, Finland</conf-loc></conference></article-meta></front><body><sec sec-type="intro"><title>Introduction</title><p>Many proteins bind DNA and do that in a sequence specific way. These DNA-binding proteins include transcription factors (TF), among others, which have an important function in regulating gene expression by affecting transcription and chromatin state. Given the fundamental role of TFs in cellular processes and difficulty in measuring their binding sites, computational analysis of binding sites can provide tremendous help in understanding complex regulatory mechanisms [<xref ref-type="bibr" rid="B1">1</xref>]. The DNA preference of DNA-binding proteins can be modelled with different computational methods [<xref ref-type="bibr" rid="B2">2</xref>]. All methods require known binding sites or data from biological experiments, such as gene expression profiling, chromatin immunoprecipitation followed by sequencing (ChIP-seq), protein binding microarrays (PBM) or systematic evolution of ligands by exponential enrichment (SELEX) followed by sequencing, to build the models.</p><p>The modeling method which has become the major paradigm is the position specific weight matrix (PWM) model [<xref ref-type="bibr" rid="B3">3</xref>,<xref ref-type="bibr" rid="B4">4</xref>]. Position weight matrices are constructed by using sequences from an experiment or automatically aligning binding sites within longer sequences [<xref ref-type="bibr" rid="B5">5</xref>,<xref ref-type="bibr" rid="B6">6</xref>]. The number of each base is calculated in each position of the alignment, and then each base is assigned a score based on the counts. This way each position treats the nucleotides independently from the other positions: the score is based only on the frequency of the base in that certain position. Consequently, PWMs have been criticized that they might lose some important dependencies between nearby nucleotides, but PWMs provide a very easy and intuitive modeling framework and, moreover, thousands of different PWMs exist in several databases [<xref ref-type="bibr" rid="B7">7</xref>,<xref ref-type="bibr" rid="B8">8</xref>].</p><p><italic>k</italic>-mer models offer a different perspective in DNA-protein interactions. In <italic>k</italic>-mer models each <italic>k</italic>-mer, a nucleotide sequence of length <italic>k</italic>, is given a score that describes the binding affinity of the protein towards the <italic>k</italic>-mer. The score can be assigned to each <italic>k</italic>-mer for example by utilizing the distribution of <italic>k</italic>-mers in the binding sites, or solving the score using modeling approaches. <italic>k</italic>-mer models help in capturing depedencies between the nucleotide positions in the binding. A recent comparison study demonstrated that <italic>k</italic>-mer models can provide equal or better performance compared to traditional PWM models, particularly so on high-throughout PBM data [<xref ref-type="bibr" rid="B2">2</xref>]. On contrary, performance of <italic>k</italic>-mer models relative to PWM-based methods was found to severely degrade when tested on <italic>in vivo </italic>data.</p><p>Here we focus on the <italic>k</italic>-mer model by Annala <italic>et al. </italic>[<xref ref-type="bibr" rid="B9">9</xref>] which was ranked the first in the original DREAM5 challenge and was among the two most accurate methods in a comprehensive comparison by Weirauch <italic>et al. </italic>[<xref ref-type="bibr" rid="B2">2</xref>]. We apply the <italic>k</italic>-mer model to 15 proteins for which SELEX data is available [<xref ref-type="bibr" rid="B10">10</xref>] and, thus, report performance of the <italic>k</italic>-mer model on another high-throughput protocol complementary to PBM data used in [<xref ref-type="bibr" rid="B2">2</xref>]. The performance is also compared against PWM models derived from the same data.</p><p>To improve robustness and generalization of <italic>k</italic>-mer models we studied different feature selection strategies for choosing the most informative <italic>k</italic>-mers to the model. The first feature selection strategies were to select the most frequent or the most enriched <italic>k</italic>-mers. In addition we implemented the cross-validation scheme for choosing the <italic>k</italic>-mers. Finally, the <italic>k</italic>-mer model trained with SELEX data was evaluated on ChIP-seq data with varying results.</p></sec><sec sec-type="materials|methods"><title>Materials and methods</title><sec><title>SELEX data</title><p>In SELEX protocol [<xref ref-type="bibr" rid="B10">10</xref>] the experiment starts with a large pool of all different DNA-sequences of fixed length. The protein of interest is introduced into the pool, and the protein will recognize and bind its target sequences. Then an antibody is added to the solution and the protein-DNA complexes are immunoprecipitated. The bound sequences are then amplified and sequenced. The process is repeated by using the bound sequences as a new initial pool of sequences. The protocol produces massive amount of bound sequences (reads) for different iterations which can be used to construct models for describing DNA-protein interactions as well as to evaluate their predictive performance.</p><p>The SELEX data from [<xref ref-type="bibr" rid="B10">10</xref>] contained enriched reads from the SELEX process for different enrichment cycles. To account for non-specific carryover of DNA from previous cycles, a specially designed multinomial method was used to construct PWMs from high-throughput SELEX data in [<xref ref-type="bibr" rid="B10">10</xref>]. The obtained PWMs were found to be very similar with those obtained by the standard MEME [<xref ref-type="bibr" rid="B5">5</xref>], thus providing us a good benchmark. The PWM-models were however constructed using only one cycle, and only that cycle was taken into consideration in the performance comparison between <italic>k</italic>-mer and PWM models. Cycles that were included consisted between 9000 and 175000 reads with average being 62000 reads. These sets of reads were randomly divided to training and test sets with ratio 7:3. The training set was used for training the <italic>k</italic>-mer model, and the test set was used in the performance analysis and comparisons. Random reads (negative set) were generated to match the number of enriched reads for each training and testing set. The choice to use uniformly random sequences instead of picking random genomic locations was justified by the fact that in the original experiment the initial pool of nucleotide sequences included evenly all 14-mers. The model performance and comparison was evaluated by a classification task where the test data set consisting of enriched reads not used in the training and equal amount of random reads.</p></sec><sec><title>Description of the linear <bold><italic>k</italic></bold>-mer model</title><p>The linear <italic>k</italic>-mer model presented by Annala <italic>et al. </italic>[<xref ref-type="bibr" rid="B9">9</xref>] assigns an affinity score to each <italic>k</italic>-mer and the total binding affinity to a certain sequence is the sum of the binding affinities of the <italic>k</italic>-mers in that sequence. In the original work this affinity was measured by signal strength of a spot in PBM. The affinity score for the <italic>k</italic>-mers can be solved from the linear model <italic>Ax </italic>= <italic>b </italic>+ <italic>ε</italic>, where <italic>A </italic>is the design matrix, <italic>x </italic>is the affinities (column vector), <italic>b </italic>is signal strength and <italic>E </italic>represents noise. In the design matrix each row represents a probe and columns are the variables, the <italic>k</italic>-mers. The <italic>A<sub>ij </sub></italic>element of the matrix is one if the <italic>k</italic>-mer corresponding to the <italic>j</italic>th column can be found in the <italic>i</italic>th probe, zero otherwise.</p><p>The <italic>k</italic>-mer model is trained by solving the linear equation <italic>Ax </italic>= <italic>b</italic>. In the original application for PBMs the <italic>b </italic>vector resulting from the multiplication consisted of probe intensities. For SELEX experiments no intensity data is measured but instead the <italic>b </italic>vector is a binary vector where one denotes a detected enriched (<italic>i.e</italic>. bound) read and zero denotes a random read (unbound). The training requires both bound and unbound sequence reads in order to produce useful predictive models. In this model training scheme the reads were treated as equally important, which might not be true, as the binding affinity of a protein will vary between individual reads. However, given sufficient sequencing depth, the substantial number of reads could compensate this assumption because strongly bound <italic>k</italic>-mers would occur more frequently in the data.</p></sec><sec><title>The choice of <bold><italic>k</italic></bold>-mers</title><p>A key question with the linear <italic>k</italic>-mer model is which <italic>k</italic>-mers should be included into the model. To begin with, we chose two ways to do this. First way was to choose the most common <italic>k</italic>-mers in the whole data, the second way was to choose the most enriched <italic>k</italic>-mers. The enriched <italic>k</italic>-mers were picked from the <italic>k</italic>-mer table which lists the counts of each <italic>k</italic>-mer in the set of bound reads and random unbound reads and sorts them based on either the difference or fold change of those counts. The higher the difference or fold change, the more enriched the <italic>k</italic>-mer in question is. The standard cross-validation scheme was also implemented. First we started with a predefined number of most frequent <italic>k</italic>-mers, and then we started removing those <italic>k</italic>-mers from the model, whose removal improved the results most or had the smallest negative effect in 10-fold cross-validation classification within the training set, essentially implementing a wrapper type of feature selection technique. The number of <italic>k</italic>-mers in the beginning should not be too high, because cross-validation computation gets heavy easily. Note that, to avoid the selection bias [<xref ref-type="bibr" rid="B11">11</xref>], the final prediction performance for the cross-validated model was assessed on the separate testing data used neither during the model training nor during the feature selection.</p></sec></sec><sec sec-type="results"><title>Results</title><p>The <italic>k</italic>-mer models trained as explained above and the PWM models taken from Jolma et al. [<xref ref-type="bibr" rid="B10">10</xref>] were used in classifying the reads in the testing set. Note that the PWM models were derived using a combination of training and testing data, which may slightly positively influence the PWM results. The classification using PWMs was conducted by scanning the reads in the testing set with the given model and the maximum of the scores was assigned to that read. The reads were classified using threshold that was found optimal in the training set. Classification accuracy is estimated as the proportion of reads that are classified correctly. The confidence intervals are normal approximation intervals.</p><p>The classification accuracies together with 95% confidence intervals are shown in Figure <xref ref-type="fig" rid="F1">1</xref>. The <italic>k</italic>- mer model clearly outperforms the PWM models as can be seen from the figure. In the first bars the accuracy of the <italic>k</italic>-mer model is the accuracy using the optimal number of most frequent <italic>k</italic>-mers. The average classification accuracy is 71% for the <italic>k</italic>-mer model and 62% for the PWM models. In the second and third bars the accuracy is obtained with <italic>k</italic>-mer model using the most enriched <italic>k</italic>-mers. Surprisingly <italic>k</italic>-mer model performance is better, when choosing the most frequent <italic>k</italic>-mers instead of the most enriched <italic>k</italic>-mers. This might indicate that assigning high affinity scores to <italic>k</italic>-mers responsible for binding as well as giving negative affinity scores to frequent <italic>k</italic>-mers that are not part of the binding are important.</p><fig id="F1" position="float"><label>Figure 1</label><caption><p><bold>Classification accuracies of the <bold><italic>k</italic></bold>-mer model and PWM models</bold>. The test set classification accuracy is plotted for 3 different <italic>k</italic>-mer model approaches and the PWM model. The 95% normal approximation confidence intervals are plotted on top of each bar.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-1"></graphic></fig><p>The effect of the number of <italic>k</italic>-mers in the model is shown in Figure <xref ref-type="fig" rid="F2">2</xref>. The average classification accuracy increases sharply when <italic>k</italic>-mers are added to the model, and the accuracy reaches its maximum at about 600 <italic>k</italic>-mers. Using more than 600 <italic>k</italic>-mers has little effect on results. However, for individual proteins the classification accuracy seems to peak at around 600 or 1500 <italic>k</italic>-mers. In later cycles the data can be classified with great accuracy by using only one <italic>k</italic>-mer. For protein XBP1 it suffices to include only <italic>k</italic>-mer ACGT to the model, and the data can be classified with accuracy of 91%. It is worth noting, that the <italic>k</italic>-mer is its reverse complemented and can be therefore detected from both strands.</p><fig id="F2" position="float"><label>Figure 2</label><caption><p><bold>Classification accuracy as a function of number of <bold><italic>k</italic></bold>-mers in the model</bold>. The test set classification accuracy plotted as a function of number of <italic>k</italic>-mers in the model. The 95% normal approximation confidence intervals are plotted around each curve as grey area.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-2"></graphic></fig><sec><title>Selection of <bold><italic>k</italic></bold>-mers improves prediction accuracy</title><p>The cross-validation gives a slight improvement to the results. If we start with for example 200 <italic>k</italic>-mers, we can end up in a set of 100 <italic>k</italic>-mers which is better than taking just the 100 most frequent <italic>k</italic>-mers--the main motivation for feature selection in discriminatory analysis. The change in classification accuracy, when starting with 100 <italic>k</italic>-mers, is shown for four proteins in Figure <xref ref-type="fig" rid="F3">3</xref>. The blue line represents the accuracy in the 10-fold cross-validation within the training set, and red line is the classification accuracy in the testing set. Green line represents the classification accuracy, when the model is trained using the equal number of <italic>k</italic>-mers that are most frequent in the data. Horizontal axis represents the number of <italic>k</italic>-mers in the model.</p><fig id="F3" position="float"><label>Figure 3</label><caption><p><bold>Illustration of classification accuracy using the CV-scheme for four proteins</bold>. The test set classification accuracy as a function of number of <italic>k</italic>-mers, when the <italic>k</italic>-mers are chosen with the CV-scheme. The CV is started from 100 most frequent <italic>k</italic>-mers. The 95% normal approximation confidence intervals are plotted around the curves.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-3"></graphic></fig><p>The binding prediction results vary between proteins. For example for PRDM1 and HSF2 the cross-validation scheme can choose <italic>k</italic>-mers producing much better classification results. The data can be classified reliably with only seven <italic>k</italic>-mers (HSF2). With FOXJ3 taking the most frequent <italic>k</italic>-mers is equally good as choosing the <italic>k</italic>-mers with cross-validation. This means that for those proteins the most frequent <italic>k</italic>-mers are truly the most important ones in classification. On the other hand, for example for HSF2, for which the results are better using the cross-validation, the most important <italic>k</italic>-mers are somewhat longer and therefore also less frequent <italic>k</italic>-mers.</p><p>Cross-validation starting with higher number of <italic>k</italic>-mers yields similar results (Figure <xref ref-type="fig" rid="F4">4</xref>). Sometimes the most frequent <italic>k</italic>-mers yield equal or even slightly better results than cross-validation scheme, but for some proteins the cross-validation introduces great advantages. For example for HSF2 the cross-validation clearly improves results as the classification accuracy is maximized around 150 <italic>k</italic>-mers and difference between the cross-validated and the enrichment method increases even more for lower smaller number of <italic>k</italic>-mers. Moreover, especially for smaller number of <italic>k</italic>-mers the cross-validated feature selection provides significantly better results than the standard approach (Figures <xref ref-type="fig" rid="F3">3</xref> and <xref ref-type="fig" rid="F4">4</xref>).</p><fig id="F4" position="float"><label>Figure 4</label><caption><p><bold>Illustration of classification accuracy using the CV-scheme when starting with greater number of <bold><italic>k</italic></bold>-mers</bold>. The test set classification accuracy as a function of number of <italic>k</italic>-mers, when the <italic>k</italic>-mers are chosen with the CV-scheme. The CV is started from 450 and 600 most frequent <italic>k</italic>-mers for proteins HSF2 and FOXJ3. The 95% normal approximation confidence intervals are plotted around the curves.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-4"></graphic></fig><p>It was also investigated how the chosen SELEX enrichment cycle affects the classification accuracy. The results are plotted in Figure <xref ref-type="fig" rid="F5">5</xref>. It is clear that the performance is higher at later cycles. This is quite intuitive because the set of sequences should get more homogenous as the enrichment process proceeds. It is also noticeable how classifying the first SELEX rounds is much more difficult than the later cycles. At later cycles there is also much more variability in the results between proteins. The round used in the performance comparison is marked with a star.</p><fig id="F5" position="float"><label>Figure 5</label><caption><p><bold>Classification accuracy in different enrichment cycles</bold>. The classification accuracy from different SELEX enrichment cycles for five proteins. The 95% normal approximation confidence intervals are plotted around the accuracies.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-5"></graphic></fig><p>The cross-validation scheme does not choose necessarily those <italic>k</italic>-mers to the model that are important originally with higher number of <italic>k</italic>-mers. It was investigated if the highest affinity <italic>k</italic>-mers would be included late in the cross-validation. There was little overlap between the ten last <italic>k</italic>-mers in the cross-validation and ten most important (highest affinity score) <italic>k</italic>-mers in the most-frequent approach. Nevertheless the <italic>k</italic>-mers are quite similar to each other in the sense that the top <italic>k</italic>-mers in the different methods align relatively well to the PWMs: examples of FOXJ3 and HSF2 are shown in Figures <xref ref-type="fig" rid="F6">6</xref> and <xref ref-type="fig" rid="F7">7</xref>. The <italic>k</italic>-mers aligned to the logo in the left are from the latest rounds in the cross-validation and in the right the <italic>k</italic>-mers are top-affinity <italic>k</italic>-mers from the most-frequent approach. Both <italic>k</italic>-mers and their reverse complements are taken into account when aligning the top <italic>k</italic>-mers to the motifs.</p><fig id="F6" position="float"><label>Figure 6</label><caption><p><bold>Top <bold><italic>k</italic></bold>-mers aligned to FOXJ3 logo</bold>. The top <italic>k</italic>-mers chosen with the CV-scheme (left) and the top affinity <italic>k</italic>-mers from the most frequent approach (right) aligned to FOXJ3 sequence logo.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-6"></graphic></fig><fig id="F7" position="float"><label>Figure 7</label><caption><p><bold>Top <bold><italic>k</italic></bold>-mers aligned to HSF2 logo</bold>. The top <italic>k</italic>-mers chosen with the CV-scheme (left) and the top affinity <italic>k</italic>-mers from the most frequent approach (right) aligned to HSF2 sequence logo.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-7"></graphic></fig></sec><sec><title>Application to ChIP-seq data</title><p>We also assessed the applicability of the linear <italic>k</italic>-mer model, trained on SELEX data, to ChIP-seq data. That is, the model was still trained with the SELEX data but performance was evaluated using data from ChIP-seq experiments. This required finding suitable data sets for some of the proteins for which SELEX data was produced in [<xref ref-type="bibr" rid="B10">10</xref>]. Three data sets, for proteins RFX3, NFATC1 and GATA1, were used. Data for RFX3 and NFATC1 is described in Jolma <italic>et al. </italic>[<xref ref-type="bibr" rid="B10">10</xref>] and data for GATA1 was taken from the ENCODE data set. Each data set contained true peaks supposedly bound by the protein of interest, and negative peaks, locations that were chosen randomly and are most likely unbound by the protein.</p><p>Area under the curve (AUC) metric was chosen for performance evaluation: the classification threshold previously identified from SELEX data might not be applicable to other types of data, because the reads to be classified are longer. AUC reports the probability that a true bound sequence will score higher than a random non-bound sequence. AUC of 1 corresponds to perfect discovery of the true peaks without making any false positive predictions, and AUC of 0.5 can be reached by classifying the peaks randomly. Mann-Whitney confidence intervals are suitable for estimating the confidence interval of the AUC metric [<xref ref-type="bibr" rid="B12">12</xref>]. From each data set AUC was calculated using the entire peak regions that can span several hundred nucleotides and for the centers of each peak (50nt and 100nt regions).</p><p>For protein NFATC1 AUC is plotted against the number of <italic>k</italic>-mers in the model in Figure <xref ref-type="fig" rid="F8">8</xref>. As can be seen the AUC stays quite steadily in the range of 0.5 meaning that the <italic>k</italic>-mer model fails to distinquish between true peaks and negative peaks.</p><fig id="F8" position="float"><label>Figure 8</label><caption><p><bold>AUC as a function of number of <bold><italic>k</italic></bold>-mers in the model for two NFATC1 ChIP-seq samples</bold>. (left) AUCs when the true binding sites are taken to be within 100 nucleotides around the summit of the ChIP-seq peak. (right) The same as (left) except each binding site is taken to be the whole ChIP-seq peak region. The 95% Mann-Whitney confidence intervals plotted around the curves.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-8"></graphic></fig><p>For GATA1 the results are more interesting, as in some cases the <italic>k</italic>-mer model provides great predictive power. The ChIP-seq peaks can be quite accurately distinguished from false peaks as the AUC rises close to 0.9 with relatively high number of <italic>k</italic>-mers in the model (Figure <xref ref-type="fig" rid="F9">9</xref>, left). The summits in turn can be distinquished from false peaks by using only small number of <italic>k</italic>-mers selected with the cross-validation scheme (Figure <xref ref-type="fig" rid="F9">9</xref>, right). There is however somewhat controversial phenomenon of AUC dropping significantly below 0.5 when using greater number of <italic>k</italic>-mers in the model. Whether its a property of the data or the <italic>k</italic>-mer model remains to be investigated. Nevertheless, taken together, our results indicate that the previously reported poor performance of the <italic>k</italic>-mer model on <italic>in vivo </italic>data can be improved using careful feature selection strategies.</p><fig id="F9" position="float"><label>Figure 9</label><caption><p><bold>AUC as a function of number of <bold><italic>k</italic></bold>-mers in the model for GATA1 ChIP-seq samples</bold>. AUC as a function of number of <italic>k</italic>-mers for GATA1, when <italic>k</italic>-mers are selected with the CV-scheme. In left figure (red) the AUC is plotted when using the most frequent <italic>k</italic>-mers. In right figure the AUC is calculated when only the centers of the ChIP-seq peaks are used. The 95% Mann-Whitney confidence intervals plotted around the curves.</p></caption><graphic xlink:href="1471-2105-14-S10-S2-9"></graphic></fig></sec></sec><sec sec-type="discussion"><title>Discussion</title><p>Feature selection in <italic>k</italic>-mer models can be approached in many different ways. Our results indicate that it is preferable to include the most frequent <italic>k</italic>-mers in the model instead of choosing the most enriched ones. This approach inevitably favours shorter <italic>k</italic>-mers because they are statistically more likely to appear frequently in sequence data. We also observed that, for SELEX data, increasing the number of <italic>k</italic>-mers in the model does not improve predictions after certain point (about 600 <italic>k</italic>-mers). This point most likely changes depending on which kind of data the model is applied to.</p><p>The number of <italic>k</italic>-mers can be reduced without great negative effects in prediction accuracy using the standard cross-validation scheme. Consequently, this decreases the number of parameters which need to be estimated for the <italic>k</italic>-mer models making them more attractive and applicable binding prediction model.</p><p>Although it is quite clear that the <italic>k</italic>-mer model outperforms widely used PWM-models within the SELEX data set, the performance of the <italic>k</italic>-mer model with <italic>in vivo </italic>data still remains an open question. For two data sets (proteins NFATC1 and RFX3) the <italic>k</italic>-mer model failed to distinquish between true binding sites and unbound sites. For GATA1 the results however seem very promising as the AUC of close to 0.9 was reached.</p></sec><sec><title>Competing interests</title><p>The authors declare that they have no competing interests.</p></sec><sec><title>Authors' contributions</title><p>JK analyzed the data, collected results and wrote the paper. HL designed and supervised the study and wrote the paper.</p></sec></body><back><sec><title>Acknowledgements</title><p>This work was supported by the EU ERASysBio ERA-NET, Emil Aaltonen Foundation and Academy of Finland Center of Excellence in Systems Immunology and Physiology.</p></sec><sec><title>Declarations</title><p>The publication costs for this article are covered by Academy of Finland Center of Excellence in Systems Immunology and Physiology.</p><p>This article has been published as part of BMC Bioinformatics Volume 14 Supplement 10, 2013: Selected articles from the 10th International Workshop on Computational Systems Biology (WCSB) 2013: Bioinformatics. The full contents of the supplement are available online at <ext-link ext-link-type="uri" xlink:href="http://www.biomedcentral.com/bmcbioinformatics/supplements/14/S10">http://www.biomedcentral.com/bmcbioinformatics/supplements/14/S10</ext-link></p></sec><ref-list><ref id="B1"><mixed-citation publication-type="journal"><name><surname>Wasserman</surname><given-names>W</given-names></name><name><surname>Sandelin</surname><given-names>A</given-names></name><article-title>Applied bioinformatics for the identification of regulatory elements</article-title><source>Nature Reviews Genetics</source><year>2004</year><volume>5</volume><issue>4</issue><fpage>276</fpage><lpage>87</lpage><pub-id pub-id-type="doi">10.1038/nrg1315</pub-id><pub-id pub-id-type="pmid">15131651</pub-id></mixed-citation></ref><ref id="B2"><mixed-citation publication-type="journal"><name><surname>Weirauch</surname><given-names>M</given-names></name><name><surname>Cote</surname><given-names>A</given-names></name><name><surname>Norel</surname><given-names>R</given-names></name><name><surname>Annala</surname><given-names>M</given-names></name><name><surname>Zhao</surname><given-names>Y</given-names></name><name><surname>Riley</surname><given-names>T</given-names></name><name><surname>Saez-Rodriguez</surname><given-names>J</given-names></name><name><surname>Cokelaer</surname><given-names>T</given-names></name><name><surname>Vedenko</surname><given-names>A</given-names></name><name><surname>Talukder</surname><given-names>S</given-names></name><collab>DREAM5 Consortium</collab><name><surname>Bussemaker</surname><given-names>H</given-names></name><name><surname>Morris</surname><given-names>Q</given-names></name><name><surname>Bulyk</surname><given-names>M</given-names></name><name><surname>Stolovitzky</surname><given-names>G</given-names></name><name><surname>Hughes</surname><given-names>T</given-names></name><article-title>Evaluation of methods for modeling transcription factor sequence specificity</article-title><source>Nature Biotechnology</source><year>2013</year><volume>2</volume><fpage>126</fpage><lpage>34</lpage><pub-id pub-id-type="pmid">23354101</pub-id></mixed-citation></ref><ref id="B3"><mixed-citation publication-type="journal"><name><surname>Staden</surname><given-names>R</given-names></name><article-title>Computer methods to locate signals in nucleic acid sequences</article-title><source>Nucleic Acids Research</source><year>1984</year><volume>12</volume><fpage>505</fpage><lpage>519</lpage><pub-id pub-id-type="doi">10.1093/nar/12.1Part2.505</pub-id><pub-id pub-id-type="pmid">6364039</pub-id></mixed-citation></ref><ref id="B4"><mixed-citation publication-type="journal"><name><surname>Stormo</surname><given-names>G</given-names></name><name><surname>Schneider</surname><given-names>T</given-names></name><name><surname>Gold</surname><given-names>L</given-names></name><name><surname>Ehrenfeucht</surname><given-names>A</given-names></name><article-title>Use of the 'Perceptron' algorithm to distinguish translational initiation sites in E. coli</article-title><source>Nucleic Acids Res</source><year>1982</year><volume>10</volume><fpage>2997</fpage><lpage>3011</lpage><pub-id pub-id-type="doi">10.1093/nar/10.9.2997</pub-id><pub-id pub-id-type="pmid">7048259</pub-id></mixed-citation></ref><ref id="B5"><mixed-citation publication-type="journal"><name><surname>Bailey</surname><given-names>T</given-names></name><name><surname>Elkan</surname><given-names>C</given-names></name><article-title>Fitting a mixture model by expectation maximization to discover motifs in biopolymers</article-title><source>Proceedings of the Second International Conference on Intelligent Systems for Molecular Biology</source><year>1994</year><volume>2</volume><fpage>28</fpage><lpage>36</lpage><pub-id pub-id-type="pmid">7584402</pub-id></mixed-citation></ref><ref id="B6"><mixed-citation publication-type="journal"><name><surname>Lawrence</surname><given-names>C</given-names></name><name><surname>Altschul</surname><given-names>S</given-names></name><name><surname>Boguski</surname><given-names>M</given-names></name><name><surname>Liu</surname><given-names>J</given-names></name><name><surname>Neuwald</surname><given-names>A</given-names></name><name><surname>Wootton</surname><given-names>J</given-names></name><article-title>Detecting subtle sequence signals: A Gibbs sampling strategy for multiple alignment</article-title><source>Science</source><year>1993</year><volume>262</volume><fpage>208</fpage><lpage>214</lpage><pub-id pub-id-type="doi">10.1126/science.8211139</pub-id><pub-id pub-id-type="pmid">8211139</pub-id></mixed-citation></ref><ref id="B7"><mixed-citation publication-type="journal"><name><surname>Wingender</surname><given-names>E</given-names></name><name><surname>Chen</surname><given-names>X</given-names></name><name><surname>Hehl</surname><given-names>R</given-names></name><name><surname>Karas</surname><given-names>H</given-names></name><name><surname>Liebich</surname><given-names>I</given-names></name><name><surname>Matys</surname><given-names>V</given-names></name><name><surname>Meinhardt</surname><given-names>T</given-names></name><name><surname>Prüss</surname><given-names>M</given-names></name><name><surname>Reuter</surname><given-names>I</given-names></name><name><surname>Schacherer</surname><given-names>F</given-names></name><article-title>TRANSFAC: an integrated system for gene expression regulation</article-title><source>Nucleic Acids Research</source><year>2000</year><volume>28</volume><fpage>316</fpage><lpage>319</lpage><pub-id pub-id-type="doi">10.1093/nar/28.1.316</pub-id><pub-id pub-id-type="pmid">10592259</pub-id></mixed-citation></ref><ref id="B8"><mixed-citation publication-type="journal"><name><surname>Sandelin</surname><given-names>A</given-names></name><name><surname>Alkema</surname><given-names>W</given-names></name><name><surname>Engström</surname><given-names>P</given-names></name><name><surname>Wasserman</surname><given-names>W</given-names></name><name><surname>Lenhard</surname><given-names>B</given-names></name><article-title>JASPAR: an open-access database for eukaryotic transcription factor binding profiles</article-title><source>Nucleic Acids Research</source><year>2004</year><volume>32</volume><fpage>D91</fpage><lpage>D94</lpage><pub-id pub-id-type="doi">10.1093/nar/gkh012</pub-id><pub-id pub-id-type="pmid">14681366</pub-id></mixed-citation></ref><ref id="B9"><mixed-citation publication-type="journal"><name><surname>Annala</surname><given-names>M</given-names></name><name><surname>Laurila</surname><given-names>K</given-names></name><name><surname>Lähdesmäki</surname><given-names>H</given-names></name><name><surname>Nykter</surname><given-names>M</given-names></name><article-title>A linear model for transcription factor binding affinity prediction in protein binding microarrays</article-title><source>PLoS ONE</source><year>2011</year><volume>6</volume><issue>5</issue><fpage>e20059</fpage><pub-id pub-id-type="doi">10.1371/journal.pone.0020059</pub-id><pub-id pub-id-type="pmid">21637853</pub-id></mixed-citation></ref><ref id="B10"><mixed-citation publication-type="journal"><name><surname>Jolma</surname><given-names>A</given-names></name><name><surname>Kivioja</surname><given-names>T</given-names></name><name><surname>Toivonen</surname><given-names>J</given-names></name><name><surname>Cheng</surname><given-names>L</given-names></name><name><surname>Wei</surname><given-names>G</given-names></name><name><surname>Enge</surname><given-names>M</given-names></name><name><surname>Taipale</surname><given-names>M</given-names></name><name><surname>Vaquerizas</surname><given-names>J</given-names></name><name><surname>Yan</surname><given-names>J</given-names></name><name><surname>Sillanpää</surname><given-names>M</given-names></name><name><surname>Bonke</surname><given-names>M</given-names></name><name><surname>Palin</surname><given-names>K</given-names></name><name><surname>Talukder</surname><given-names>S</given-names></name><name><surname>Hughes</surname><given-names>T</given-names></name><name><surname>Luscombe</surname><given-names>N</given-names></name><name><surname>Ukkonen</surname><given-names>E</given-names></name><name><surname>Taipale</surname><given-names>J</given-names></name><article-title>Multiplexed massively parallel SELEX for characterization of human transcription factor binding specificities</article-title><source>Genome Res</source><year>2010</year><volume>20</volume><issue>6</issue><fpage>861</fpage><lpage>73</lpage><pub-id pub-id-type="doi">10.1101/gr.100552.109</pub-id><pub-id pub-id-type="pmid">20378718</pub-id></mixed-citation></ref><ref id="B11"><mixed-citation publication-type="journal"><name><surname>Ambroise</surname><given-names>C</given-names></name><name><surname>McLachlan</surname><given-names>G</given-names></name><article-title>Selection bias in gene extraction on the basis of microarray gene-expression data</article-title><source>PNAS</source><year>2002</year><volume>99</volume><issue>10</issue><fpage>6562</fpage><lpage>6566</lpage><pub-id pub-id-type="doi">10.1073/pnas.102102699</pub-id><pub-id pub-id-type="pmid">11983868</pub-id></mixed-citation></ref><ref id="B12"><mixed-citation publication-type="journal"><name><surname>Gengsheng</surname><given-names>Q</given-names></name><name><surname>Hotilovac</surname><given-names>L</given-names></name><article-title>Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test</article-title><source>Stat Methods Med Res</source><year>2008</year><volume>7</volume><fpage>207</fpage><lpage>221</lpage><pub-id pub-id-type="pmid">18426855</pub-id></mixed-citation></ref></ref-list></back></pmc></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Sante/explor/MersV1/Data/Pmc/Corpus
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 0009480 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd -nk 0009480 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Sante
|area= MersV1
|flux= Pmc
|étape= Corpus
|type= RBID
|clé=
|texte=
}}
| This area was generated with Dilib version V0.6.33. |  |