Analyzing big datasets of genomic sequences: fast and scalable collection of k-mer statistics
Identifieur interne : 000280 ( Pmc/Curation ); précédent : 000279; suivant : 000281Analyzing big datasets of genomic sequences: fast and scalable collection of k-mer statistics
Auteurs : Umberto Ferraro Petrillo [Italie] ; Mara Sorella [Italie] ; Giuseppe Cattaneo [Italie] ; Raffaele Giancarlo [Italie] ; Simona E. Rombo [Italie]Source :
- BMC Bioinformatics [ 1471-2105 ] ; 2019.
Abstract
Distributed approaches based on the MapReduce programming paradigm have started to be proposed in the Bioinformatics domain, due to the large amount of data produced by the next-generation sequencing techniques. However, the use of MapReduce and related Big Data technologies and frameworks (e.g., Apache Hadoop and Spark) does not necessarily produce satisfactory results, in terms of both efficiency and effectiveness. We discuss how the development of distributed and Big Data management technologies has affected the analysis of large datasets of biological sequences. Moreover, we show how the choice of different parameter configurations and the careful engineering of the software with respect to the specific framework under consideration may be crucial in order to achieve good performance, especially on very large amounts of data. We choose
One of the most relevant contributions of FastKmer is the introduction of a module for balancing the statistics aggregation workload over the nodes of a computing cluster, in order to overcome data skew while allowing for a full exploitation of the underlying distributed architecture. We also present the results of a comparative experimental analysis showing that our approach is currently the fastest among the ones based on Big Data technologies, while exhibiting a very good scalability.
We provide evidence that the usage of technologies such as Hadoop or Spark for the analysis of big datasets of biological sequences is productive only if the architectural details and the peculiar aspects of the considered framework are carefully taken into account for the algorithm design and implementation.
Url:
DOI: 10.1186/s12859-019-2694-8
PubMed: 30999863
PubMed Central: 6471689
Links toward previous steps (curation, corpus...)
- to stream Pmc, to step Corpus: Pour aller vers cette notice dans l'étape Curation :000280
Links to Exploration step
PMC:6471689Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Analyzing big datasets of genomic sequences: fast and scalable collection of <italic>k</italic>-mer statistics</title><author><name sortKey="Ferraro Petrillo, Umberto" sort="Ferraro Petrillo, Umberto" uniqKey="Ferraro Petrillo U" first="Umberto" last="Ferraro Petrillo">Umberto Ferraro Petrillo</name><affiliation wicri:level="1"><nlm:aff id="Aff1"><institution-wrap><institution-id institution-id-type="GRID">grid.7841.a</institution-id><institution>Dipartimento di Scienze Statistiche, Università di Roma - La Sapienza,</institution></institution-wrap>Rome, 00185 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Rome</wicri:regionArea></affiliation></author><author><name sortKey="Sorella, Mara" sort="Sorella, Mara" uniqKey="Sorella M" first="Mara" last="Sorella">Mara Sorella</name><affiliation wicri:level="1"><nlm:aff id="Aff2"><institution-wrap><institution-id institution-id-type="GRID">grid.7841.a</institution-id><institution>Dipartimento di Ingegneria Informatica, Automatica e Gestionale, Università di Roma - La Sapienza,</institution></institution-wrap>Rome, 00185 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Rome</wicri:regionArea></affiliation></author><author><name sortKey="Cattaneo, Giuseppe" sort="Cattaneo, Giuseppe" uniqKey="Cattaneo G" first="Giuseppe" last="Cattaneo">Giuseppe Cattaneo</name><affiliation wicri:level="1"><nlm:aff id="Aff3"><institution-wrap><institution-id institution-id-type="ISNI">0000 0004 1937 0335</institution-id><institution-id institution-id-type="GRID">grid.11780.3f</institution-id><institution>Dipartimento di Informatica, Università di Salerno,</institution></institution-wrap>Fisciano (SA), 84084 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Fisciano (SA)</wicri:regionArea></affiliation></author><author><name sortKey="Giancarlo, Raffaele" sort="Giancarlo, Raffaele" uniqKey="Giancarlo R" first="Raffaele" last="Giancarlo">Raffaele Giancarlo</name><affiliation wicri:level="1"><nlm:aff id="Aff4"><institution-wrap><institution-id institution-id-type="ISNI">0000 0004 1762 5517</institution-id><institution-id institution-id-type="GRID">grid.10776.37</institution-id><institution>Dipartimento di Matematica ed Informatica, Università di Palermo,</institution></institution-wrap>Palermo, 90133 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Palermo</wicri:regionArea></affiliation></author><author><name sortKey="Rombo, Simona E" sort="Rombo, Simona E" uniqKey="Rombo S" first="Simona E." last="Rombo">Simona E. Rombo</name><affiliation wicri:level="1"><nlm:aff id="Aff4"><institution-wrap><institution-id institution-id-type="ISNI">0000 0004 1762 5517</institution-id><institution-id institution-id-type="GRID">grid.10776.37</institution-id><institution>Dipartimento di Matematica ed Informatica, Università di Palermo,</institution></institution-wrap>Palermo, 90133 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Palermo</wicri:regionArea></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">PMC</idno><idno type="pmid">30999863</idno><idno type="pmc">6471689</idno><idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC6471689</idno><idno type="RBID">PMC:6471689</idno><idno type="doi">10.1186/s12859-019-2694-8</idno><date when="2019">2019</date><idno type="wicri:Area/Pmc/Corpus">000280</idno><idno type="wicri:explorRef" wicri:stream="Pmc" wicri:step="Corpus" wicri:corpus="PMC">000280</idno><idno type="wicri:Area/Pmc/Curation">000280</idno><idno type="wicri:explorRef" wicri:stream="Pmc" wicri:step="Curation">000280</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Analyzing big datasets of genomic sequences: fast and scalable collection of <italic>k</italic>-mer statistics</title><author><name sortKey="Ferraro Petrillo, Umberto" sort="Ferraro Petrillo, Umberto" uniqKey="Ferraro Petrillo U" first="Umberto" last="Ferraro Petrillo">Umberto Ferraro Petrillo</name><affiliation wicri:level="1"><nlm:aff id="Aff1"><institution-wrap><institution-id institution-id-type="GRID">grid.7841.a</institution-id><institution>Dipartimento di Scienze Statistiche, Università di Roma - La Sapienza,</institution></institution-wrap>Rome, 00185 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Rome</wicri:regionArea></affiliation></author><author><name sortKey="Sorella, Mara" sort="Sorella, Mara" uniqKey="Sorella M" first="Mara" last="Sorella">Mara Sorella</name><affiliation wicri:level="1"><nlm:aff id="Aff2"><institution-wrap><institution-id institution-id-type="GRID">grid.7841.a</institution-id><institution>Dipartimento di Ingegneria Informatica, Automatica e Gestionale, Università di Roma - La Sapienza,</institution></institution-wrap>Rome, 00185 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Rome</wicri:regionArea></affiliation></author><author><name sortKey="Cattaneo, Giuseppe" sort="Cattaneo, Giuseppe" uniqKey="Cattaneo G" first="Giuseppe" last="Cattaneo">Giuseppe Cattaneo</name><affiliation wicri:level="1"><nlm:aff id="Aff3"><institution-wrap><institution-id institution-id-type="ISNI">0000 0004 1937 0335</institution-id><institution-id institution-id-type="GRID">grid.11780.3f</institution-id><institution>Dipartimento di Informatica, Università di Salerno,</institution></institution-wrap>Fisciano (SA), 84084 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Fisciano (SA)</wicri:regionArea></affiliation></author><author><name sortKey="Giancarlo, Raffaele" sort="Giancarlo, Raffaele" uniqKey="Giancarlo R" first="Raffaele" last="Giancarlo">Raffaele Giancarlo</name><affiliation wicri:level="1"><nlm:aff id="Aff4"><institution-wrap><institution-id institution-id-type="ISNI">0000 0004 1762 5517</institution-id><institution-id institution-id-type="GRID">grid.10776.37</institution-id><institution>Dipartimento di Matematica ed Informatica, Università di Palermo,</institution></institution-wrap>Palermo, 90133 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Palermo</wicri:regionArea></affiliation></author><author><name sortKey="Rombo, Simona E" sort="Rombo, Simona E" uniqKey="Rombo S" first="Simona E." last="Rombo">Simona E. Rombo</name><affiliation wicri:level="1"><nlm:aff id="Aff4"><institution-wrap><institution-id institution-id-type="ISNI">0000 0004 1762 5517</institution-id><institution-id institution-id-type="GRID">grid.10776.37</institution-id><institution>Dipartimento di Matematica ed Informatica, Università di Palermo,</institution></institution-wrap>Palermo, 90133 Italy</nlm:aff><country xml:lang="fr">Italie</country><wicri:regionArea>Palermo</wicri:regionArea></affiliation></author></analytic><series><title level="j">BMC Bioinformatics</title><idno type="eISSN">1471-2105</idno><imprint><date when="2019">2019</date></imprint></series></biblStruct></sourceDesc></fileDesc><profileDesc><textClass></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en"><sec><title>Background</title><p>Distributed approaches based on the MapReduce programming paradigm have started to be proposed in the Bioinformatics domain, due to the large amount of data produced by the next-generation sequencing techniques. However, the use of MapReduce and related Big Data technologies and frameworks (e.g., Apache Hadoop and Spark) does not necessarily produce satisfactory results, in terms of both efficiency and effectiveness. We discuss how the development of distributed and Big Data management technologies has affected the analysis of large datasets of biological sequences. Moreover, we show how the choice of different parameter configurations and the careful engineering of the software with respect to the specific framework under consideration may be crucial in order to achieve good performance, especially on very large amounts of data. We choose <italic>k</italic>-mers counting as a case study for our analysis, and Spark as the framework to implement FastKmer, a novel approach for the extraction of <italic>k</italic>-mer statistics from large collection of biological sequences, with arbitrary values of <italic>k</italic>.</p></sec><sec><title>Results</title><p>One of the most relevant contributions of FastKmer is the introduction of a module for balancing the statistics aggregation workload over the nodes of a computing cluster, in order to overcome data skew while allowing for a full exploitation of the underlying distributed architecture. We also present the results of a comparative experimental analysis showing that our approach is currently the fastest among the ones based on Big Data technologies, while exhibiting a very good scalability.</p></sec><sec><title>Conclusions</title><p>We provide evidence that the usage of technologies such as Hadoop or Spark for the analysis of big datasets of biological sequences is productive only if the architectural details and the peculiar aspects of the considered framework are carefully taken into account for the algorithm design and implementation.</p></sec></div></front><back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Ferraro Petrillo, U" uniqKey="Ferraro Petrillo U">U Ferraro Petrillo</name></author><author><name sortKey="Roscigno, G" uniqKey="Roscigno G">G Roscigno</name></author><author><name sortKey="Cattaneo, G" uniqKey="Cattaneo G">G Cattaneo</name></author><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cattaneo, G" uniqKey="Cattaneo G">G Cattaneo</name></author><author><name sortKey="Ferraro Petrillo, U" uniqKey="Ferraro Petrillo U">U Ferraro Petrillo</name></author><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author><author><name sortKey="Roscigno, G" uniqKey="Roscigno G">G Roscigno</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cattaneo, G" uniqKey="Cattaneo G">G Cattaneo</name></author><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author><author><name sortKey="Piotto, S" uniqKey="Piotto S">S Piotto</name></author><author><name sortKey="Ferraro Petrillo, U" uniqKey="Ferraro Petrillo U">U Ferraro Petrillo</name></author><author><name sortKey="Roscigno, G" uniqKey="Roscigno G">G Roscigno</name></author><author><name sortKey="Di Biasi, L" uniqKey="Di Biasi L">L Di Biasi</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Ferraro Petrillo, U" uniqKey="Ferraro Petrillo U">U Ferraro Petrillo</name></author><author><name sortKey="Guerra, C" uniqKey="Guerra C">C Guerra</name></author><author><name sortKey="Pizzi, C" uniqKey="Pizzi C">C Pizzi</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Zhou, W" uniqKey="Zhou W">W Zhou</name></author><author><name sortKey="Li, R" uniqKey="Li R">R Li</name></author><author><name sortKey="Yuan, S" uniqKey="Yuan S">S Yuan</name></author><author><name sortKey="Liu, C" uniqKey="Liu C">C Liu</name></author><author><name sortKey="Yao, S" uniqKey="Yao S">S Yao</name></author><author><name sortKey="Luo, J" uniqKey="Luo J">J Luo</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Benoit, G" uniqKey="Benoit G">G Benoit</name></author><author><name sortKey="Peterlongo, P" uniqKey="Peterlongo P">P Peterlongo</name></author><author><name sortKey="Mariadassou, M" uniqKey="Mariadassou M">M Mariadassou</name></author><author><name sortKey="Drezen, E" uniqKey="Drezen E">E Drezen</name></author><author><name sortKey="Schbath, S" uniqKey="Schbath S">S Schbath</name></author><author><name sortKey="Lavenier, D" uniqKey="Lavenier D">D Lavenier</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author><author><name sortKey="Scaturro, D" uniqKey="Scaturro D">D Scaturro</name></author><author><name sortKey="Utro, F" uniqKey="Utro F">F Utro</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author><author><name sortKey="Rombo, Se" uniqKey="Rombo S">SE Rombo</name></author><author><name sortKey="Utro, F" uniqKey="Utro F">F Utro</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Lo Bosco, G" uniqKey="Lo Bosco G">G Lo Bosco</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Nordstrom, Kjv" uniqKey="Nordstrom K">KJV Nordstrom</name></author><author><name sortKey="Albani, Mc" uniqKey="Albani M">MC Albani</name></author><author><name sortKey="James, Gv" uniqKey="James G">GV James</name></author><author><name sortKey="Gutjahr, C" uniqKey="Gutjahr C">C Gutjahr</name></author><author><name sortKey="Gutjahr, C" uniqKey="Gutjahr C">C Gutjahr</name></author><author><name sortKey="Turck, F" uniqKey="Turck F">F Turck</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Pinello, L" uniqKey="Pinello L">L Pinello</name></author><author><name sortKey="Lo Bosco, G" uniqKey="Lo Bosco G">G Lo Bosco</name></author><author><name sortKey="Hanlon, B" uniqKey="Hanlon B">B Hanlon</name></author><author><name sortKey="Yuan, Gc" uniqKey="Yuan G">GC Yuan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Utro, F" uniqKey="Utro F">F Utro</name></author><author><name sortKey="Di Benedetto, V" uniqKey="Di Benedetto V">V Di Benedetto</name></author><author><name sortKey="Corona, Dfv" uniqKey="Corona D">DFV Corona</name></author><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author><author><name sortKey="Rombo, Se" uniqKey="Rombo S">SE Rombo</name></author><author><name sortKey="Utro, F" uniqKey="Utro F">F Utro</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author><author><name sortKey="Rombo, Se" uniqKey="Rombo S">SE Rombo</name></author><author><name sortKey="Utro, F" uniqKey="Utro F">F Utro</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author><author><name sortKey="Rombo, Se" uniqKey="Rombo S">SE Rombo</name></author><author><name sortKey="Utro, F" uniqKey="Utro F">F Utro</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Ferraro Petrillo, U" uniqKey="Ferraro Petrillo U">U Ferraro Petrillo</name></author><author><name sortKey="Roscigno, G" uniqKey="Roscigno G">G Roscigno</name></author><author><name sortKey="Cattaneo, G" uniqKey="Cattaneo G">G Cattaneo</name></author><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Dean, J" uniqKey="Dean J">J Dean</name></author><author><name sortKey="Ghemawat, S" uniqKey="Ghemawat S">S Ghemawat</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Siretskiy, A" uniqKey="Siretskiy A">A Siretskiy</name></author><author><name sortKey="Sundqvist, T" uniqKey="Sundqvist T">T Sundqvist</name></author><author><name sortKey="Voznesenskiy, M" uniqKey="Voznesenskiy M">M Voznesenskiy</name></author><author><name sortKey="Spjuth, O" uniqKey="Spjuth O">O Spjuth</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Zaharia, M" uniqKey="Zaharia M">M Zaharia</name></author><author><name sortKey="Chowdhury, M" uniqKey="Chowdhury M">M Chowdhury</name></author><author><name sortKey="Franklin, Mj" uniqKey="Franklin M">MJ Franklin</name></author><author><name sortKey="Shenker, S" uniqKey="Shenker S">S Shenker</name></author><author><name sortKey="Stoica, I" uniqKey="Stoica I">I Stoica</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Horwege, S" uniqKey="Horwege S">S Horwege</name></author><author><name sortKey="Lindner, S" uniqKey="Lindner S">S Lindner</name></author><author><name sortKey="Boden, M" uniqKey="Boden M">M Boden</name></author><author><name sortKey="Hatje, K" uniqKey="Hatje K">K Hatje</name></author><author><name sortKey="Kollmar, M" uniqKey="Kollmar M">M Kollmar</name></author><author><name sortKey="Leimeister, Ca" uniqKey="Leimeister C">CA Leimeister</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Leimeister, Ca" uniqKey="Leimeister C">CA Leimeister</name></author><author><name sortKey="Boden, M" uniqKey="Boden M">M Boden</name></author><author><name sortKey="Horwege, S" uniqKey="Horwege S">S Horwege</name></author><author><name sortKey="Lindner, S" uniqKey="Lindner S">S Lindner</name></author><author><name sortKey="Morgenstern, B" uniqKey="Morgenstern B">B Morgenstern</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Dean, J" uniqKey="Dean J">J Dean</name></author><author><name sortKey="Ghemawat, S" uniqKey="Ghemawat S">S Ghemawat</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Vavilapalli, Vk" uniqKey="Vavilapalli V">VK Vavilapalli</name></author><author><name sortKey="Murthy, Ac" uniqKey="Murthy A">AC Murthy</name></author><author><name sortKey="Douglas, C" uniqKey="Douglas C">C Douglas</name></author><author><name sortKey="Agarwal, S" uniqKey="Agarwal S">S Agarwal</name></author><author><name sortKey="Konar, M" uniqKey="Konar M">M Konar</name></author><author><name sortKey="Evans, R" uniqKey="Evans R">R Evans</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Shvachko, K" uniqKey="Shvachko K">K Shvachko</name></author><author><name sortKey="Kuang, H" uniqKey="Kuang H">H Kuang</name></author><author><name sortKey="Radia, S" uniqKey="Radia S">S Radia</name></author><author><name sortKey="Chansler, R" uniqKey="Chansler R">R Chansler</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Gropp, W" uniqKey="Gropp W">W Gropp</name></author><author><name sortKey="Lusk, E" uniqKey="Lusk E">E Lusk</name></author><author><name sortKey="Skjellum, A" uniqKey="Skjellum A">A Skjellum</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Geist, A" uniqKey="Geist A">A Geist</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Coghlan, S" uniqKey="Coghlan S">S Coghlan</name></author><author><name sortKey="Yelick, K" uniqKey="Yelick K">K Yelick</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Olston, C" uniqKey="Olston C">C Olston</name></author><author><name sortKey="Reed, B" uniqKey="Reed B">B Reed</name></author><author><name sortKey="Srivastava, U" uniqKey="Srivastava U">U Srivastava</name></author><author><name sortKey="Kumar, R" uniqKey="Kumar R">R Kumar</name></author><author><name sortKey="Tomkins, A" uniqKey="Tomkins A">A Tomkins</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hindman, B" uniqKey="Hindman B">B Hindman</name></author><author><name sortKey="Konwinski, A" uniqKey="Konwinski A">A Konwinski</name></author><author><name sortKey="Zaharia, M" uniqKey="Zaharia M">M Zaharia</name></author><author><name sortKey="Ghodsi, A" uniqKey="Ghodsi A">A Ghodsi</name></author><author><name sortKey="Joseph, Ad" uniqKey="Joseph A">AD Joseph</name></author><author><name sortKey="Katz, Rh" uniqKey="Katz R">RH Katz</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Vinga, S" uniqKey="Vinga S">S Vinga</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Compeau, Pec" uniqKey="Compeau P">PEC Compeau</name></author><author><name sortKey="Pevzner, P" uniqKey="Pevzner P">P Pevzner</name></author><author><name sortKey="Tesler, G" uniqKey="Tesler G">G Tesler</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Deorowicz, S" uniqKey="Deorowicz S">S Deorowicz</name></author><author><name sortKey="Kokot, M" uniqKey="Kokot M">M Kokot</name></author><author><name sortKey="Grabowski, S" uniqKey="Grabowski S">S Grabowski</name></author><author><name sortKey="Debudaj Grabysz, A" uniqKey="Debudaj Grabysz A">A Debudaj-Grabysz</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kokot, M" uniqKey="Kokot M">M Kokot</name></author><author><name sortKey="Dlugosz, M" uniqKey="Dlugosz M">M Długosz</name></author><author><name sortKey="Deorowicz, S" uniqKey="Deorowicz S">S Deorowicz</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Nordberg, H" uniqKey="Nordberg H">H Nordberg</name></author><author><name sortKey="Bhatia, K" uniqKey="Bhatia K">K Bhatia</name></author><author><name sortKey="Wang, K" uniqKey="Wang K">K Wang</name></author><author><name sortKey="Wang, Z" uniqKey="Wang Z">Z Wang</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Roberts, M" uniqKey="Roberts M">M Roberts</name></author><author><name sortKey="Hayes, W" uniqKey="Hayes W">W Hayes</name></author><author><name sortKey="Hunt, Br" uniqKey="Hunt B">BR Hunt</name></author><author><name sortKey="Mount, Sm" uniqKey="Mount S">SM Mount</name></author><author><name sortKey="Yorke, Ja" uniqKey="Yorke J">JA Yorke</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Deorowicz, S" uniqKey="Deorowicz S">S Deorowicz</name></author><author><name sortKey="Kokot, M" uniqKey="Kokot M">M Kokot</name></author><author><name sortKey="Grabowski, S" uniqKey="Grabowski S">S Grabowski</name></author><author><name sortKey="Debudaj Grabysz, A" uniqKey="Debudaj Grabysz A">A Debudaj-Grabysz</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Cock, Pj" uniqKey="Cock P">PJ Cock</name></author><author><name sortKey="Fields, Cj" uniqKey="Fields C">CJ Fields</name></author><author><name sortKey="Goto, N" uniqKey="Goto N">N Goto</name></author><author><name sortKey="Heuer, Ml" uniqKey="Heuer M">ML Heuer</name></author><author><name sortKey="Rice, Pm" uniqKey="Rice P">PM Rice</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Li, Y" uniqKey="Li Y">Y Li</name></author><author><name sortKey="Kamousi, P" uniqKey="Kamousi P">P Kamousi</name></author><author><name sortKey="Han, F" uniqKey="Han F">F Han</name></author><author><name sortKey="Yang, S" uniqKey="Yang S">S Yang</name></author><author><name sortKey="Yan, X" uniqKey="Yan X">X Yan</name></author><author><name sortKey="Suri, S" uniqKey="Suri S">S Suri</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Li, Y" uniqKey="Li Y">Y Li</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Ferraro Petrillo, U" uniqKey="Ferraro Petrillo U">U Ferraro Petrillo</name></author><author><name sortKey="Roscigno, G" uniqKey="Roscigno G">G Roscigno</name></author><author><name sortKey="Cattaneo, G" uniqKey="Cattaneo G">G Cattaneo</name></author><author><name sortKey="Giancarlo, R" uniqKey="Giancarlo R">R Giancarlo</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hess, M" uniqKey="Hess M">M Hess</name></author><author><name sortKey="Sczyrba, A" uniqKey="Sczyrba A">A Sczyrba</name></author><author><name sortKey="Egan, R" uniqKey="Egan R">R Egan</name></author><author><name sortKey="Kim, Tw" uniqKey="Kim T">TW Kim</name></author><author><name sortKey="Chokhawala, H" uniqKey="Chokhawala H">H Chokhawala</name></author><author><name sortKey="Schroth, G" uniqKey="Schroth G">G Schroth</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Johnson, Ds" uniqKey="Johnson D">DS Johnson</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Graham, Rl" uniqKey="Graham R">RL Graham</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kokot, M" uniqKey="Kokot M">M Kokot</name></author><author><name sortKey="Dlugosz, M" uniqKey="Dlugosz M">M Długosz</name></author><author><name sortKey="Deorowicz, S" uniqKey="Deorowicz S">S Deorowicz</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Ousterhout, K" uniqKey="Ousterhout K">K Ousterhout</name></author><author><name sortKey="Rasti, R" uniqKey="Rasti R">R Rasti</name></author><author><name sortKey="Ratnasamy, S" uniqKey="Ratnasamy S">S Ratnasamy</name></author><author><name sortKey="Shenker, S" uniqKey="Shenker S">S Shenker</name></author><author><name sortKey="Chun, Bg" uniqKey="Chun B">BG Chun</name></author><author><name sortKey="Icsi, V" uniqKey="Icsi V">V ICSI</name></author></analytic></biblStruct></listBibl></div1></back></TEI><pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">BMC Bioinformatics</journal-id><journal-id journal-id-type="iso-abbrev">BMC Bioinformatics</journal-id><journal-title-group><journal-title>BMC Bioinformatics</journal-title></journal-title-group><issn pub-type="epub">1471-2105</issn><publisher><publisher-name>BioMed Central</publisher-name><publisher-loc>London</publisher-loc></publisher></journal-meta><article-meta><article-id pub-id-type="pmid">30999863</article-id><article-id pub-id-type="pmc">6471689</article-id><article-id pub-id-type="publisher-id">2694</article-id><article-id pub-id-type="doi">10.1186/s12859-019-2694-8</article-id><article-categories><subj-group subj-group-type="heading"><subject>Research</subject></subj-group></article-categories><title-group><article-title>Analyzing big datasets of genomic sequences: fast and scalable collection of <italic>k</italic>-mer statistics</article-title></title-group><contrib-group><contrib contrib-type="author" equal-contrib="yes"><name><surname>Ferraro Petrillo</surname><given-names>Umberto</given-names></name><address><email>umberto.ferraro@uniroma1.it</email></address><xref ref-type="aff" rid="Aff1">1</xref></contrib><contrib contrib-type="author" equal-contrib="yes"><name><surname>Sorella</surname><given-names>Mara</given-names></name><address><email>sorella@diag.uniroma1.it</email></address><xref ref-type="aff" rid="Aff2">2</xref></contrib><contrib contrib-type="author"><name><surname>Cattaneo</surname><given-names>Giuseppe</given-names></name><address><email>cattaneo@unisa.it</email></address><xref ref-type="aff" rid="Aff3">3</xref></contrib><contrib contrib-type="author" corresp="yes"><name><surname>Giancarlo</surname><given-names>Raffaele</given-names></name><address><email>raffaele.giancarlo@unipa.it</email></address><xref ref-type="aff" rid="Aff4">4</xref></contrib><contrib contrib-type="author"><name><surname>Rombo</surname><given-names>Simona E.</given-names></name><address><email>simona.rombo@unipa.it</email></address><xref ref-type="aff" rid="Aff4">4</xref></contrib><aff id="Aff1"><label>1</label><institution-wrap><institution-id institution-id-type="GRID">grid.7841.a</institution-id><institution>Dipartimento di Scienze Statistiche, Università di Roma - La Sapienza,</institution></institution-wrap>Rome, 00185 Italy</aff><aff id="Aff2"><label>2</label><institution-wrap><institution-id institution-id-type="GRID">grid.7841.a</institution-id><institution>Dipartimento di Ingegneria Informatica, Automatica e Gestionale, Università di Roma - La Sapienza,</institution></institution-wrap>Rome, 00185 Italy</aff><aff id="Aff3"><label>3</label><institution-wrap><institution-id institution-id-type="ISNI">0000 0004 1937 0335</institution-id><institution-id institution-id-type="GRID">grid.11780.3f</institution-id><institution>Dipartimento di Informatica, Università di Salerno,</institution></institution-wrap>Fisciano (SA), 84084 Italy</aff><aff id="Aff4"><label>4</label><institution-wrap><institution-id institution-id-type="ISNI">0000 0004 1762 5517</institution-id><institution-id institution-id-type="GRID">grid.10776.37</institution-id><institution>Dipartimento di Matematica ed Informatica, Università di Palermo,</institution></institution-wrap>Palermo, 90133 Italy</aff></contrib-group><pub-date pub-type="epub"><day>18</day><month>4</month><year>2019</year></pub-date><pub-date pub-type="pmc-release"><day>18</day><month>4</month><year>2019</year></pub-date><pub-date pub-type="collection"><year>2019</year></pub-date><volume>20</volume><issue>Suppl 4</issue><elocation-id>138</elocation-id><permissions><copyright-statement>© The Author(s) 2019</copyright-statement><license license-type="OpenAccess"><license-p><bold>Open Access</bold> This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0/">http://creativecommons.org/licenses/by/4.0/</ext-link>), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/publicdomain/zero/1.0/">http://creativecommons.org/publicdomain/zero/1.0/</ext-link>) applies to the data made available in this article, unless otherwise stated.</license-p></license></permissions><abstract id="Abs1"><sec><title>Background</title><p>Distributed approaches based on the MapReduce programming paradigm have started to be proposed in the Bioinformatics domain, due to the large amount of data produced by the next-generation sequencing techniques. However, the use of MapReduce and related Big Data technologies and frameworks (e.g., Apache Hadoop and Spark) does not necessarily produce satisfactory results, in terms of both efficiency and effectiveness. We discuss how the development of distributed and Big Data management technologies has affected the analysis of large datasets of biological sequences. Moreover, we show how the choice of different parameter configurations and the careful engineering of the software with respect to the specific framework under consideration may be crucial in order to achieve good performance, especially on very large amounts of data. We choose <italic>k</italic>-mers counting as a case study for our analysis, and Spark as the framework to implement FastKmer, a novel approach for the extraction of <italic>k</italic>-mer statistics from large collection of biological sequences, with arbitrary values of <italic>k</italic>.</p></sec><sec><title>Results</title><p>One of the most relevant contributions of FastKmer is the introduction of a module for balancing the statistics aggregation workload over the nodes of a computing cluster, in order to overcome data skew while allowing for a full exploitation of the underlying distributed architecture. We also present the results of a comparative experimental analysis showing that our approach is currently the fastest among the ones based on Big Data technologies, while exhibiting a very good scalability.</p></sec><sec><title>Conclusions</title><p>We provide evidence that the usage of technologies such as Hadoop or Spark for the analysis of big datasets of biological sequences is productive only if the architectural details and the peculiar aspects of the considered framework are carefully taken into account for the algorithm design and implementation.</p></sec></abstract><kwd-group xml:lang="en"><title>Keywords</title><kwd>Distributed computing</kwd><kwd>Apache Spark</kwd><kwd>k-mer counting</kwd><kwd>Performance evaluation</kwd></kwd-group><conference xlink:href="http://www.igst.it/nettab/2017/"><conf-name>The 2017 Network Tools and Applications in Biology (NETTAB) Workshop</conf-name><conf-acronym>NETTAB 2017</conf-acronym><conf-loc>Palermo, Italy</conf-loc><conf-date>16-18 October 2017</conf-date></conference><custom-meta-group><custom-meta><meta-name>issue-copyright-statement</meta-name><meta-value>© The Author(s) 2019</meta-value></custom-meta></custom-meta-group></article-meta></front></pmc></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Sante/explor/MersV1/Data/Pmc/Curation
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000280 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Curation/biblio.hfd -nk 000280 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Sante
|area= MersV1
|flux= Pmc
|étape= Curation
|type= RBID
|clé= PMC:6471689
|texte= Analyzing big datasets of genomic sequences: fast and scalable collection of k-mer statistics
}}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Curation/RBID.i -Sk "pubmed:30999863" \
| HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Curation/biblio.hfd \
| NlmPubMed2Wicri -a MersV1
| This area was generated with Dilib version V0.6.33. |  |