The Widening Gulf between Genomics Data Generation and Consumption: A Practical Guide to Big Data Transfer Technology
Identifieur interne : 000066 ( Main/Merge ); précédent : 000065; suivant : 000067The Widening Gulf between Genomics Data Generation and Consumption: A Practical Guide to Big Data Transfer Technology
Auteurs : Frank A. Feltus [États-Unis] ; Joseph R. Breen [États-Unis] ; Juan Deng [États-Unis] ; Ryan S. Izard [États-Unis] ; Christopher A. Konger [États-Unis] ; Walter B. Ligon [États-Unis] ; Don Preuss [États-Unis] ; Kuang-Ching Wang [États-Unis]Source :
- Bioinformatics and Biology Insights [ 1177-9322 ] ; 2015.
Abstract
In the last decade, high-throughput DNA sequencing has become a disruptive technology and pushed the life sciences into a distributed ecosystem of sequence data producers and consumers. Given the power of genomics and declining sequencing costs, biology is an emerging “Big Data” discipline that will soon enter the exabyte data range when all subdisciplines are combined. These datasets must be transferred across commercial and research networks in creative ways since sending data without thought can have serious consequences on data processing time frames. Thus, it is imperative that biologists, bioinformaticians, and information technology engineers recalibrate data processing paradigms to fit this emerging reality. This review attempts to provide a snapshot of Big Data transfer across networks, which is often overlooked by many biologists. Specifically, we discuss four key areas: 1) data transfer networks, protocols, and applications; 2) data transfer security including encryption, access, firewalls, and the Science DMZ; 3) data flow control with software-defined networking; and 4) data storage, staging, archiving and access. A primary intention of this article is to orient the biologist in key aspects of the data transfer process in order to frame their genomics-oriented needs to enterprise IT professionals.
Url:
DOI: 10.4137/BBI.S28988
PubMed: 26568680
PubMed Central: 4636112
Links toward previous steps (curation, corpus...)
- to stream Pmc, to step Corpus: 000093
- to stream Pmc, to step Curation: 000093
- to stream Pmc, to step Checkpoint: 000060
- to stream Ncbi, to step Merge: 000690
- to stream Ncbi, to step Curation: 000690
- to stream Ncbi, to step Checkpoint: 000690
Links to Exploration step
PMC:4636112Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">The Widening Gulf between Genomics Data Generation and Consumption: A Practical Guide to Big Data Transfer Technology</title><author><name sortKey="Feltus, Frank A" sort="Feltus, Frank A" uniqKey="Feltus F" first="Frank A." last="Feltus">Frank A. Feltus</name><affiliation wicri:level="2"><nlm:aff id="af1-bbi-suppl.1-2015-009">Department of Genetics and Biochemistry, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Genetics and Biochemistry, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Breen, Joseph R" sort="Breen, Joseph R" uniqKey="Breen J" first="Joseph R." last="Breen">Joseph R. Breen</name><affiliation wicri:level="2"><nlm:aff id="af2-bbi-suppl.1-2015-009">University of Utah Center for High Performance Computing, Salt Lake City, UT, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>University of Utah Center for High Performance Computing, Salt Lake City, UT</wicri:regionArea><placeName><region type="state">Utah</region></placeName></affiliation></author><author><name sortKey="Deng, Juan" sort="Deng, Juan" uniqKey="Deng J" first="Juan" last="Deng">Juan Deng</name><affiliation wicri:level="2"><nlm:aff id="af3-bbi-suppl.1-2015-009">Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Electrical and Computer Engineering, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Izard, Ryan S" sort="Izard, Ryan S" uniqKey="Izard R" first="Ryan S." last="Izard">Ryan S. Izard</name><affiliation wicri:level="2"><nlm:aff id="af3-bbi-suppl.1-2015-009">Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Electrical and Computer Engineering, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Konger, Christopher A" sort="Konger, Christopher A" uniqKey="Konger C" first="Christopher A." last="Konger">Christopher A. Konger</name><affiliation wicri:level="2"><nlm:aff id="af4-bbi-suppl.1-2015-009">Clemson Computing and Information Technology, Clemson University, Anderson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Clemson Computing and Information Technology, Clemson University, Anderson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Ligon, Walter B" sort="Ligon, Walter B" uniqKey="Ligon W" first="Walter B." last="Ligon">Walter B. Ligon</name><affiliation wicri:level="2"><nlm:aff id="af3-bbi-suppl.1-2015-009">Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Electrical and Computer Engineering, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Preuss, Don" sort="Preuss, Don" uniqKey="Preuss D" first="Don" last="Preuss">Don Preuss</name><affiliation wicri:level="2"><nlm:aff id="af5-bbi-suppl.1-2015-009">NIH/NLM/NCBI, Bethesda, MD, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>NIH/NLM/NCBI, Bethesda, MD</wicri:regionArea><placeName><region type="state">Maryland</region></placeName></affiliation></author><author><name sortKey="Wang, Kuang Ching" sort="Wang, Kuang Ching" uniqKey="Wang K" first="Kuang-Ching" last="Wang">Kuang-Ching Wang</name><affiliation wicri:level="2"><nlm:aff id="af3-bbi-suppl.1-2015-009">Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Electrical and Computer Engineering, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">PMC</idno><idno type="pmid">26568680</idno><idno type="pmc">4636112</idno><idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4636112</idno><idno type="RBID">PMC:4636112</idno><idno type="doi">10.4137/BBI.S28988</idno><date when="2015">2015</date><idno type="wicri:Area/Pmc/Corpus">000093</idno><idno type="wicri:Area/Pmc/Curation">000093</idno><idno type="wicri:Area/Pmc/Checkpoint">000060</idno><idno type="wicri:Area/Ncbi/Merge">000690</idno><idno type="wicri:Area/Ncbi/Curation">000690</idno><idno type="wicri:Area/Ncbi/Checkpoint">000690</idno><idno type="wicri:Area/Main/Merge">000066</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">The Widening Gulf between Genomics Data Generation and Consumption: A Practical Guide to Big Data Transfer Technology</title><author><name sortKey="Feltus, Frank A" sort="Feltus, Frank A" uniqKey="Feltus F" first="Frank A." last="Feltus">Frank A. Feltus</name><affiliation wicri:level="2"><nlm:aff id="af1-bbi-suppl.1-2015-009">Department of Genetics and Biochemistry, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Genetics and Biochemistry, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Breen, Joseph R" sort="Breen, Joseph R" uniqKey="Breen J" first="Joseph R." last="Breen">Joseph R. Breen</name><affiliation wicri:level="2"><nlm:aff id="af2-bbi-suppl.1-2015-009">University of Utah Center for High Performance Computing, Salt Lake City, UT, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>University of Utah Center for High Performance Computing, Salt Lake City, UT</wicri:regionArea><placeName><region type="state">Utah</region></placeName></affiliation></author><author><name sortKey="Deng, Juan" sort="Deng, Juan" uniqKey="Deng J" first="Juan" last="Deng">Juan Deng</name><affiliation wicri:level="2"><nlm:aff id="af3-bbi-suppl.1-2015-009">Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Electrical and Computer Engineering, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Izard, Ryan S" sort="Izard, Ryan S" uniqKey="Izard R" first="Ryan S." last="Izard">Ryan S. Izard</name><affiliation wicri:level="2"><nlm:aff id="af3-bbi-suppl.1-2015-009">Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Electrical and Computer Engineering, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Konger, Christopher A" sort="Konger, Christopher A" uniqKey="Konger C" first="Christopher A." last="Konger">Christopher A. Konger</name><affiliation wicri:level="2"><nlm:aff id="af4-bbi-suppl.1-2015-009">Clemson Computing and Information Technology, Clemson University, Anderson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Clemson Computing and Information Technology, Clemson University, Anderson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Ligon, Walter B" sort="Ligon, Walter B" uniqKey="Ligon W" first="Walter B." last="Ligon">Walter B. Ligon</name><affiliation wicri:level="2"><nlm:aff id="af3-bbi-suppl.1-2015-009">Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Electrical and Computer Engineering, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author><author><name sortKey="Preuss, Don" sort="Preuss, Don" uniqKey="Preuss D" first="Don" last="Preuss">Don Preuss</name><affiliation wicri:level="2"><nlm:aff id="af5-bbi-suppl.1-2015-009">NIH/NLM/NCBI, Bethesda, MD, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>NIH/NLM/NCBI, Bethesda, MD</wicri:regionArea><placeName><region type="state">Maryland</region></placeName></affiliation></author><author><name sortKey="Wang, Kuang Ching" sort="Wang, Kuang Ching" uniqKey="Wang K" first="Kuang-Ching" last="Wang">Kuang-Ching Wang</name><affiliation wicri:level="2"><nlm:aff id="af3-bbi-suppl.1-2015-009">Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA.</nlm:aff><country xml:lang="fr" wicri:curation="lc">États-Unis</country><wicri:regionArea>Department of Electrical and Computer Engineering, Clemson University, Clemson, SC</wicri:regionArea><placeName><region type="state">Caroline du Sud</region></placeName></affiliation></author></analytic><series><title level="j">Bioinformatics and Biology Insights</title><idno type="eISSN">1177-9322</idno><imprint><date when="2015">2015</date></imprint></series></biblStruct></sourceDesc></fileDesc><profileDesc><textClass></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en"><p>In the last decade, high-throughput DNA sequencing has become a disruptive technology and pushed the life sciences into a distributed ecosystem of sequence data producers and consumers. Given the power of genomics and declining sequencing costs, biology is an emerging “Big Data” discipline that will soon enter the exabyte data range when all subdisciplines are combined. These datasets must be transferred across commercial and research networks in creative ways since sending data without thought can have serious consequences on data processing time frames. Thus, it is imperative that biologists, bioinformaticians, and information technology engineers recalibrate data processing paradigms to fit this emerging reality. This review attempts to provide a snapshot of Big Data transfer across networks, which is often overlooked by many biologists. Specifically, we discuss four key areas: 1) data transfer networks, protocols, and applications; 2) data transfer security including encryption, access, firewalls, and the Science DMZ; 3) data flow control with software-defined networking; and 4) data storage, staging, archiving and access. A primary intention of this article is to orient the biologist in key aspects of the data transfer process in order to frame their genomics-oriented needs to enterprise IT professionals.</p></div></front><back><div1 type="bibliography"><listBibl><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Denoeud, F" uniqKey="Denoeud F">F Denoeud</name></author><author><name sortKey="Carretero Paulet, L" uniqKey="Carretero Paulet L">L Carretero-Paulet</name></author><author><name sortKey="Dereeper, A" uniqKey="Dereeper A">A Dereeper</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Dabydeen, Sa" uniqKey="Dabydeen S">SA Dabydeen</name></author><author><name sortKey="Kang, K" uniqKey="Kang K">K Kang</name></author><author><name sortKey="Diaz Cruz, Es" uniqKey="Diaz Cruz E">ES Díaz-Cruz</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Sanger, F" uniqKey="Sanger F">F Sanger</name></author><author><name sortKey="Coulson, Ar" uniqKey="Coulson A">AR Coulson</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kodama, Y" uniqKey="Kodama Y">Y Kodama</name></author><author><name sortKey="Shumway, M" uniqKey="Shumway M">M Shumway</name></author><author><name sortKey="Leinonen, R" uniqKey="Leinonen R">R Leinonen</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Wilks, C" uniqKey="Wilks C">C Wilks</name></author><author><name sortKey="Cline, Ms" uniqKey="Cline M">MS Cline</name></author><author><name sortKey="Weiler, E" uniqKey="Weiler E">E Weiler</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Foster, I" uniqKey="Foster I">I Foster</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Leduc, R" uniqKey="Leduc R">R LeDuc</name></author><author><name sortKey="Vaughn, M" uniqKey="Vaughn M">M Vaughn</name></author><author><name sortKey="Fonner, Jm" uniqKey="Fonner J">JM Fonner</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Goff, Sa" uniqKey="Goff S">SA Goff</name></author><author><name sortKey="Vaughn, M" uniqKey="Vaughn M">M Vaughn</name></author><author><name sortKey="Mckay, S" uniqKey="Mckay S">S McKay</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Crosswell, Lc" uniqKey="Crosswell L">LC Crosswell</name></author><author><name sortKey="Thornton, Jm" uniqKey="Thornton J">JM Thornton</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Lampa, S" uniqKey="Lampa S">S Lampa</name></author><author><name sortKey="Dahlo, M" uniqKey="Dahlo M">M Dahlo</name></author><author><name sortKey="Olason, Pi" uniqKey="Olason P">PI Olason</name></author><author><name sortKey="Hagberg, J" uniqKey="Hagberg J">J Hagberg</name></author><author><name sortKey="Spjuth, O" uniqKey="Spjuth O">O Spjuth</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Afgan, E" uniqKey="Afgan E">E Afgan</name></author><author><name sortKey="Chapman, B" uniqKey="Chapman B">B Chapman</name></author><author><name sortKey="Jadan, M" uniqKey="Jadan M">M Jadan</name></author><author><name sortKey="Franke, V" uniqKey="Franke V">V Franke</name></author><author><name sortKey="Taylor, J" uniqKey="Taylor J">J Taylor</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Dart, E" uniqKey="Dart E">E Dart</name></author><author><name sortKey="Rotman, L" uniqKey="Rotman L">L Rotman</name></author><author><name sortKey="Tierney, B" uniqKey="Tierney B">B Tierney</name></author><author><name sortKey="Hester, M" uniqKey="Hester M">M Hester</name></author><author><name sortKey="Zurawski, J" uniqKey="Zurawski J">J Zurawski</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Allcock, W" uniqKey="Allcock W">W Allcock</name></author><author><name sortKey="Bresnahan, J" uniqKey="Bresnahan J">J Bresnahan</name></author><author><name sortKey="Kettimuthu, R" uniqKey="Kettimuthu R">R Kettimuthu</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Monga, I" uniqKey="Monga I">I Monga</name></author><author><name sortKey="Pouyoul, E" uniqKey="Pouyoul E">E Pouyoul</name></author><author><name sortKey="Gu, C" uniqKey="Gu C">C Gu</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Benson, T" uniqKey="Benson T">T Benson</name></author><author><name sortKey="Akella, A" uniqKey="Akella A">A Akella</name></author><author><name sortKey="Maltz, D" uniqKey="Maltz D">D Maltz</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Mckeown, N" uniqKey="Mckeown N">N McKeown</name></author><author><name sortKey="Anderson, T" uniqKey="Anderson T">T Anderson</name></author><author><name sortKey="Balakrishnan, H" uniqKey="Balakrishnan H">H Balakrishnan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Wang, G" uniqKey="Wang G">G Wang</name></author><author><name sortKey="Ng, T" uniqKey="Ng T">T Ng</name></author><author><name sortKey="Shaikh, A" uniqKey="Shaikh A">A Shaikh</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Rosen, A" uniqKey="Rosen A">A Rosen</name></author><author><name sortKey="Wang, Kc" uniqKey="Wang K">KC Wang</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Das, A" uniqKey="Das A">A Das</name></author><author><name sortKey="Lumezanu, C" uniqKey="Lumezanu C">C Lumezanu</name></author><author><name sortKey="Zhang, Y" uniqKey="Zhang Y">Y Zhang</name></author><author><name sortKey="Singh, V" uniqKey="Singh V">V Singh</name></author><author><name sortKey="Jiang, G" uniqKey="Jiang G">G Jiang</name></author><author><name sortKey="Yu, C" uniqKey="Yu C">C Yu</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct></listBibl></div1></back></TEI></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/CyberinfraV1/Data/Main/Merge
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000066 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Merge/biblio.hfd -nk 000066 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Ticri/CIDE
|area= CyberinfraV1
|flux= Main
|étape= Merge
|type= RBID
|clé= PMC:4636112
|texte= The Widening Gulf between Genomics Data Generation and Consumption: A Practical Guide to Big Data Transfer Technology
}}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Main/Merge/RBID.i -Sk "pubmed:26568680" \
| HfdSelect -Kh $EXPLOR_AREA/Data/Main/Merge/biblio.hfd \
| NlmPubMed2Wicri -a CyberinfraV1
| This area was generated with Dilib version V0.6.25. |  |