BEETL-fastq: a searchable compressed archive for DNA reads.
Identifieur interne : 001933 ( PubMed/Corpus ); précédent : 001932; suivant : 001934BEETL-fastq: a searchable compressed archive for DNA reads.
Auteurs : Lilian Janin ; Ole Schulz-Trieglaff ; Anthony J. CoxSource :
- Bioinformatics (Oxford, England) [ 1367-4811 ] ; 2014.
English descriptors
- KwdEn :
- MESH :
- chemical : DNA.
- genetics : Neoplasms.
- methods : Data Compression, Sequence Analysis, DNA.
- Algorithms, Computer Simulation, Genome, Genome, Human, Genotype, High-Throughput Nucleotide Sequencing, Humans, Software.
Abstract
FASTQ is a standard file format for DNA sequencing data, which stores both nucleotides and quality scores. A typical sequencing study can easily generate hundreds of gigabytes of FASTQ files, while public archives such as ENA and NCBI and large international collaborations such as the Cancer Genome Atlas can accumulate many terabytes of data in this format. Compression tools such as gzip are often used to reduce the storage burden but have the disadvantage that the data must be decompressed before they can be used. Here, we present BEETL-fastq, a tool that not only compresses FASTQ-formatted DNA reads more compactly than gzip but also permits rapid search for k-mer queries within the archived sequences. Importantly, the full FASTQ record of each matching read or read pair is returned, allowing the search results to be piped directly to any of the many standard tools that accept FASTQ data as input.
DOI: 10.1093/bioinformatics/btu387
PubMed: 24950811
Links to Exploration step
pubmed:24950811Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">BEETL-fastq: a searchable compressed archive for DNA reads.</title>
<author><name sortKey="Janin, Lilian" sort="Janin, Lilian" uniqKey="Janin L" first="Lilian" last="Janin">Lilian Janin</name>
<affiliation><nlm:affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</nlm:affiliation>
</affiliation>
</author>
<author><name sortKey="Schulz Trieglaff, Ole" sort="Schulz Trieglaff, Ole" uniqKey="Schulz Trieglaff O" first="Ole" last="Schulz-Trieglaff">Ole Schulz-Trieglaff</name>
<affiliation><nlm:affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</nlm:affiliation>
</affiliation>
</author>
<author><name sortKey="Cox, Anthony J" sort="Cox, Anthony J" uniqKey="Cox A" first="Anthony J" last="Cox">Anthony J. Cox</name>
<affiliation><nlm:affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</nlm:affiliation>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PubMed</idno>
<date when="2014">2014</date>
<idno type="RBID">pubmed:24950811</idno>
<idno type="pmid">24950811</idno>
<idno type="doi">10.1093/bioinformatics/btu387</idno>
<idno type="wicri:Area/PubMed/Corpus">001933</idno>
<idno type="wicri:explorRef" wicri:stream="PubMed" wicri:step="Corpus" wicri:corpus="PubMed">001933</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en">BEETL-fastq: a searchable compressed archive for DNA reads.</title>
<author><name sortKey="Janin, Lilian" sort="Janin, Lilian" uniqKey="Janin L" first="Lilian" last="Janin">Lilian Janin</name>
<affiliation><nlm:affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</nlm:affiliation>
</affiliation>
</author>
<author><name sortKey="Schulz Trieglaff, Ole" sort="Schulz Trieglaff, Ole" uniqKey="Schulz Trieglaff O" first="Ole" last="Schulz-Trieglaff">Ole Schulz-Trieglaff</name>
<affiliation><nlm:affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</nlm:affiliation>
</affiliation>
</author>
<author><name sortKey="Cox, Anthony J" sort="Cox, Anthony J" uniqKey="Cox A" first="Anthony J" last="Cox">Anthony J. Cox</name>
<affiliation><nlm:affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</nlm:affiliation>
</affiliation>
</author>
</analytic>
<series><title level="j">Bioinformatics (Oxford, England)</title>
<idno type="eISSN">1367-4811</idno>
<imprint><date when="2014" type="published">2014</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass><keywords scheme="KwdEn" xml:lang="en"><term>Algorithms</term>
<term>Computer Simulation</term>
<term>DNA</term>
<term>Data Compression (methods)</term>
<term>Genome</term>
<term>Genome, Human</term>
<term>Genotype</term>
<term>High-Throughput Nucleotide Sequencing</term>
<term>Humans</term>
<term>Neoplasms (genetics)</term>
<term>Sequence Analysis, DNA (methods)</term>
<term>Software</term>
</keywords>
<keywords scheme="MESH" type="chemical" xml:lang="en"><term>DNA</term>
</keywords>
<keywords scheme="MESH" qualifier="genetics" xml:lang="en"><term>Neoplasms</term>
</keywords>
<keywords scheme="MESH" qualifier="methods" xml:lang="en"><term>Data Compression</term>
<term>Sequence Analysis, DNA</term>
</keywords>
<keywords scheme="MESH" xml:lang="en"><term>Algorithms</term>
<term>Computer Simulation</term>
<term>Genome</term>
<term>Genome, Human</term>
<term>Genotype</term>
<term>High-Throughput Nucleotide Sequencing</term>
<term>Humans</term>
<term>Software</term>
</keywords>
</textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en">FASTQ is a standard file format for DNA sequencing data, which stores both nucleotides and quality scores. A typical sequencing study can easily generate hundreds of gigabytes of FASTQ files, while public archives such as ENA and NCBI and large international collaborations such as the Cancer Genome Atlas can accumulate many terabytes of data in this format. Compression tools such as gzip are often used to reduce the storage burden but have the disadvantage that the data must be decompressed before they can be used. Here, we present BEETL-fastq, a tool that not only compresses FASTQ-formatted DNA reads more compactly than gzip but also permits rapid search for k-mer queries within the archived sequences. Importantly, the full FASTQ record of each matching read or read pair is returned, allowing the search results to be piped directly to any of the many standard tools that accept FASTQ data as input.</div>
</front>
</TEI>
<pubmed><MedlineCitation Status="MEDLINE" IndexingMethod="Curated" Owner="NLM"><PMID Version="1">24950811</PMID>
<DateCompleted><Year>2014</Year>
<Month>11</Month>
<Day>21</Day>
</DateCompleted>
<DateRevised><Year>2018</Year>
<Month>12</Month>
<Day>02</Day>
</DateRevised>
<Article PubModel="Print-Electronic"><Journal><ISSN IssnType="Electronic">1367-4811</ISSN>
<JournalIssue CitedMedium="Internet"><Volume>30</Volume>
<Issue>19</Issue>
<PubDate><Year>2014</Year>
<Month>Oct</Month>
</PubDate>
</JournalIssue>
<Title>Bioinformatics (Oxford, England)</Title>
<ISOAbbreviation>Bioinformatics</ISOAbbreviation>
</Journal>
<ArticleTitle>BEETL-fastq: a searchable compressed archive for DNA reads.</ArticleTitle>
<Pagination><MedlinePgn>2796-801</MedlinePgn>
</Pagination>
<ELocationID EIdType="doi" ValidYN="Y">10.1093/bioinformatics/btu387</ELocationID>
<Abstract><AbstractText Label="MOTIVATION" NlmCategory="BACKGROUND">FASTQ is a standard file format for DNA sequencing data, which stores both nucleotides and quality scores. A typical sequencing study can easily generate hundreds of gigabytes of FASTQ files, while public archives such as ENA and NCBI and large international collaborations such as the Cancer Genome Atlas can accumulate many terabytes of data in this format. Compression tools such as gzip are often used to reduce the storage burden but have the disadvantage that the data must be decompressed before they can be used. Here, we present BEETL-fastq, a tool that not only compresses FASTQ-formatted DNA reads more compactly than gzip but also permits rapid search for k-mer queries within the archived sequences. Importantly, the full FASTQ record of each matching read or read pair is returned, allowing the search results to be piped directly to any of the many standard tools that accept FASTQ data as input.</AbstractText>
<AbstractText Label="RESULTS" NlmCategory="RESULTS">We show that 6.6 terabytes of human reads in FASTQ format can be transformed into 1.7 terabytes of indexed files, from where we can search for 1, 10, 100, 1000 and a million of 30-mers in 3, 8, 14, 45 and 567 s, respectively, plus 20 ms per output read. Useful applications of the search capability are highlighted, including the genotyping of structural variant breakpoints and 'in silico pull-down' experiments in which only the reads that cover a region of interest are selectively extracted for the purposes of variant calling or visualization.</AbstractText>
<AbstractText Label="AVAILABILITY AND IMPLEMENTATION" NlmCategory="METHODS">BEETL-fastq is part of the BEETL library, available as a github repository at github.com/BEETL/BEETL.</AbstractText>
<CopyrightInformation>© The Author 2014. Published by Oxford University Press. All rights reserved. For Permissions, please e-mail: journals.permissions@oup.com.</CopyrightInformation>
</Abstract>
<AuthorList CompleteYN="Y"><Author ValidYN="Y"><LastName>Janin</LastName>
<ForeName>Lilian</ForeName>
<Initials>L</Initials>
<AffiliationInfo><Affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</Affiliation>
</AffiliationInfo>
</Author>
<Author ValidYN="Y"><LastName>Schulz-Trieglaff</LastName>
<ForeName>Ole</ForeName>
<Initials>O</Initials>
<AffiliationInfo><Affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</Affiliation>
</AffiliationInfo>
</Author>
<Author ValidYN="Y"><LastName>Cox</LastName>
<ForeName>Anthony J</ForeName>
<Initials>AJ</Initials>
<AffiliationInfo><Affiliation>Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.</Affiliation>
</AffiliationInfo>
</Author>
</AuthorList>
<Language>eng</Language>
<PublicationTypeList><PublicationType UI="D016428">Journal Article</PublicationType>
</PublicationTypeList>
<ArticleDate DateType="Electronic"><Year>2014</Year>
<Month>06</Month>
<Day>20</Day>
</ArticleDate>
</Article>
<MedlineJournalInfo><Country>England</Country>
<MedlineTA>Bioinformatics</MedlineTA>
<NlmUniqueID>9808944</NlmUniqueID>
<ISSNLinking>1367-4803</ISSNLinking>
</MedlineJournalInfo>
<ChemicalList><Chemical><RegistryNumber>9007-49-2</RegistryNumber>
<NameOfSubstance UI="D004247">DNA</NameOfSubstance>
</Chemical>
</ChemicalList>
<CitationSubset>IM</CitationSubset>
<MeshHeadingList><MeshHeading><DescriptorName UI="D000465" MajorTopicYN="N">Algorithms</DescriptorName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D003198" MajorTopicYN="N">Computer Simulation</DescriptorName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D004247" MajorTopicYN="N">DNA</DescriptorName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D044962" MajorTopicYN="N">Data Compression</DescriptorName>
<QualifierName UI="Q000379" MajorTopicYN="Y">methods</QualifierName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D016678" MajorTopicYN="N">Genome</DescriptorName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D015894" MajorTopicYN="N">Genome, Human</DescriptorName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D005838" MajorTopicYN="N">Genotype</DescriptorName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D059014" MajorTopicYN="N">High-Throughput Nucleotide Sequencing</DescriptorName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D006801" MajorTopicYN="N">Humans</DescriptorName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D009369" MajorTopicYN="N">Neoplasms</DescriptorName>
<QualifierName UI="Q000235" MajorTopicYN="Y">genetics</QualifierName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D017422" MajorTopicYN="N">Sequence Analysis, DNA</DescriptorName>
<QualifierName UI="Q000379" MajorTopicYN="Y">methods</QualifierName>
</MeshHeading>
<MeshHeading><DescriptorName UI="D012984" MajorTopicYN="N">Software</DescriptorName>
</MeshHeading>
</MeshHeadingList>
</MedlineCitation>
<PubmedData><History><PubMedPubDate PubStatus="entrez"><Year>2014</Year>
<Month>6</Month>
<Day>22</Day>
<Hour>6</Hour>
<Minute>0</Minute>
</PubMedPubDate>
<PubMedPubDate PubStatus="pubmed"><Year>2014</Year>
<Month>6</Month>
<Day>22</Day>
<Hour>6</Hour>
<Minute>0</Minute>
</PubMedPubDate>
<PubMedPubDate PubStatus="medline"><Year>2014</Year>
<Month>12</Month>
<Day>15</Day>
<Hour>6</Hour>
<Minute>0</Minute>
</PubMedPubDate>
</History>
<PublicationStatus>ppublish</PublicationStatus>
<ArticleIdList><ArticleId IdType="pubmed">24950811</ArticleId>
<ArticleId IdType="pii">btu387</ArticleId>
<ArticleId IdType="doi">10.1093/bioinformatics/btu387</ArticleId>
</ArticleIdList>
</PubmedData>
</pubmed>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Sante/explor/MersV1/Data/PubMed/Corpus
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 001933 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/PubMed/Corpus/biblio.hfd -nk 001933 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Sante |area= MersV1 |flux= PubMed |étape= Corpus |type= RBID |clé= pubmed:24950811 |texte= BEETL-fastq: a searchable compressed archive for DNA reads. }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/PubMed/Corpus/RBID.i -Sk "pubmed:24950811" \ | HfdSelect -Kh $EXPLOR_AREA/Data/PubMed/Corpus/biblio.hfd \ | NlmPubMed2Wicri -a MersV1
![]() | This area was generated with Dilib version V0.6.33. | ![]() |