DOE Joint Genome Institute

  • COVID-19
  • About
  • Phones
  • Contacts
  • Our Science
    • DOE Mission Areas
    • Bioenergy Research Centers
    • Science Programs
    • Products
    • Science Highlights
    • Scientists
    Maize can produce a cocktail of antibiotics with a handful of enzymes. (Sam Fentress, CC BY-SA 2.0)
    How Maize Makes An Antibiotic Cocktail
    Zealexins are produced in every corn variety and protect maize by fending off fungal and microbial infections using surprisingly few enzymes.

    More

    The genome of the common fiber vase or Thelephora terrestris was among those used in the study. (Francis Martin)
    From Competition to Cooperation
    By comparing 135 fungal sequenced genomes, researchers were able to carry out a broader analysis than had ever been done before to look at how saprotrophs have transitioned to the symbiotic lifestyle.

    More

    Miscanthus grasses. (Roy Kaltschmidt/Berkeley Lab)
    A Grass Model to Help Improve Giant Miscanthus
    The reference genome for M. sinensis, and the associated genomic tools, allows Miscanthus to both inform and benefit from breeding programs of related candidate bioenergy feedstock crops such as sugarcane and sorghum.

    More

  • Our Projects
    • Search JGI Projects
    • DOE Metrics/Statistics
    • Approved User Proposals
    • Legacy Projects
    Poplar (Populus trichocarpa and P. deltoides) grow in the Advanced Plant Phenotyping Laboratory (APPL) at Oak Ridge National Laboratory in Tennessee. Poplar is an important biofuel feedstock, and Populus trichocarpa is the first tree species to have its genome sequenced — a feat accomplished by JGI. (Image courtesy of Oak Ridge National Laboratory, U.S. Dept. of Energy)
    Podcast: Xiaohan Yang on A Plantiful Future
    Building off plant genomics collaborations between the JGI and Oak Ridge National Laboratory, Xiaohan Yang envisions customizing plants for the benefit of human society.

    More:

    Expansin complex with cell wall in background. (Courtesy of Daniel Cosgrove)
    Synthesizing Microbial Expansins with Unusual Activities
    Expansin proteins from diverse microbes have potential uses in deconstructing lignocellulosic biomass for conversion to renewable biofuels, nanocellulosic fibers, and commodity biochemicals.

    Read more

    High oleic pennycress. (Courtesy of Ratan Chopra)
    Pennycress – A Solution for Global Food Security, Renewable Energy and Ecosystem Benefits
    Pennycress (Thlaspi arvense) is under development as a winter annual oilseed bioenergy crop. It could produce up to 3 billion gallons of seed oil annually while reducing soil erosion and fertilizer runoff.

    Read more

  • Data & Tools
    • IMG
    • Genome Portal
    • MycoCosm
    • PhycoCosm
    • Phytozome
    • GOLD
    Artistic interpretation of CheckV assessing virus genome sequences from environmental samples. (Rendered by Zosia Rostomian​, Berkeley Lab)
    An Automated Tool for Assessing Virus Data Quality
    CheckV can be broadly utilized by the research community to gauge virus data quality and will help researchers to follow best practices and guidelines for providing the minimum amount of information for an uncultivated virus genome.

    More

    Unicellular algae in the Chlorella genus, magnified 1300x. (Andrei Savitsky)
    A One-Stop Shop for Analyzing Algal Genomes
    The PhycoCosm data portal is an interactive browser that allows algal scientists and enthusiasts to look deep into more than 100 algal genomes, compare them, and visualize supporting experimental data.

    More

    Artistic interpretation of how microbial genome sequences from the GEM catalog can help fill in gaps of knowledge about the microbes that play key roles in the Earth's microbiomes. (Rendered by Zosia Rostomian​, Berkeley Lab)
    Podcast: A Primer on Genome Mining
    In Natural Prodcast: the basics of genome mining, and how JGI researchers conducted it in IMG/ABC on thousands of metagenome-derived genomes for a Nature Biotechnology paper.

    Read more

  • User Programs
    • Calls for User Proposals
    • Special Initiatives & Programs
    • User Support
    • Submit a Proposal
    Scanning electron micrographs of diverse diatoms. (Credits: Diana Sarno, Marina Montresor, Nicole Poulsen, Gerhard Dieckmann)
    Learn About the Approved 2021 Large-Scale CSP Proposals
    A total of 27 proposals have been approved through JGI's annual Community Science Program (CSP) call. For the first time, 63 percent of the accepted proposals come from researchers who have not previously been a principal investigator on an approved JGI proposal.

    Read more

    MiddleGaylor Michael Beman UC Merced
    How to Successfully Apply for a CSP Proposal
    Reach out to JGI staff for feedback before submitting a proposal. Be sure to describe in detail what you will do with the data.

    Read more

    Click on the image or go here to watch the video "Enriching target populations for genomic analyses using HCR-FISH" from the journal Microbiome describing the research.
    How to Target a Microbial Needle within a Community Haystack
    Enabled by the JGI’s Emerging Technologies Opportunity Program, researchers have developed, tested and deployed a pipeline to first target cells from communities of uncultivated microbes, and then efficiently retrieve and characterize their genomes.

    Read more

  • News & Publications
    • News
    • Blog
    • Podcasts
    • Publications
    • Scientific Posters
    • Newsletter
    • Logos and Templates
    • Photos
    Artistic interpretation of how microbial genome sequences from the GEM catalog can help fill in gaps of knowledge about the microbes that play key roles in the Earth's microbiomes. (Rendered by Zosia Rostomian​, Berkeley Lab)
    Uncovering Novel Genomes from Earth’s Microbiomes
    A public repository of 52,515 microbial draft genomes generated from environmental samples around the world, expanding the known diversity of bacteria and archaea by 44%, is now available .

    More

    Green millet (Setaria viridis) plant collected in the wild. (Courtesy of the Kellogg lab)
    Shattering Expectations: Novel Seed Dispersal Gene Found in Green Millet
    In Nature Biotechnology, a very high quality reference Setaria viridis genome was sequenced, and for the first time in wild populations, a gene related to seed dispersal was identified.

    More

    The Brachypodium distachyon-B. stacei-B. hybridum polyploid model complex. (Illustrations credits: Juan Luis Castillo)
    The More the Merrier: Making the Case for Plant Pan-genomes
    Crop breeders have harnessed polyploidy to increase fruit and flower size, and confer stress tolerance traits. Using a Brachypodium model system, researchers have sought to learn the origins, evolution and development of plant polyploids. The work recently appeared in Nature Communications.

    Read more

Data & Tools
Home › Data & Tools › BBTools › BBTools User Guide › Clumpify Guide

Clumpify Guide

Clumpify is a tool designed to rapidly group overlapping reads into clumps. This can be used as a way to increase file compression, accelerate overlap-based assembly, or accelerate applications such as mapping or that are cache-sensitive. Clumpify can also generate consensus sequence from these clusters, though this is currently rudimentary. The clusters are not guaranteed to be overlapping; rather, they are guaranteed to share a kmer, meaning they are likely to overlap. It is designed for accurate data, meaning Illumina, Ion Torrent, or error-corrected PacBio; it will not work well with high error rate data. Even for Illumina data, quality-trimming or error-correction may be prudent.

*Notes*

Paired reads:

Clumpify supports paired reads, in which case it will clump based on read 1 only. However, it’s much more effective to treat reads as unpaired. For example, merge the reads with BBMerge, then concatenate the merged reads with the unmerged pairs, and clump them all together as unpaired.

Memory, Disk, and Phases:

Clumpify stores all sequences in memory while clumping. But it is also capable of operating in two phases, KmerSplit and KmerSort. KmerSplit will break the data up into an arbitrary number of temporary files which can then be sorted independently, and subsequently merged. As a result, Clumpify does not have a strict bound on how much memory it needs or how many sequences it can process, since the user can specify however many groups are desired to make the files arbitrarily small. This also means that the sort operation is O(N*log(N/groups)) rather than O(N*log(N)), and as groups is arbitrary, the result is O(N). If groups is set to 1, then the split phase will be skipped, and no temp files will be written, so despite the worse theoretical complexity that will typically be faster.

Compression:

Gzip compression is more efficient when similar sequences are nearby, as they can be replaced by pointers to prior copies of that sequence. So, a clumpified file will compress smaller than a randomly-ordered file. Linebreaks, headers, and quality values take up the majority of the space in a compressed clumpified file. The most efficient way to compress a sequence file, then, is to store it in fasta format with unlimited line-wrap length and replace the headers with short strings; these can be done with reformat.sh and rename.sh, if the quality values and headers are not important. Also, setting ziplevel to the max (zl=9) increases compression.

*Usage Examples*

To clumpify reads:

clumpify.sh in=reads.fq.gz out=clumped.fq.gz groups=16

This will use 16 temp files during clumpification.
To maximally compress sequence data:
rename.sh in=reads.fq out=renamed.fq prefix=x
bbmerge.sh in=renamed.fq out=merged.fq mix
clumpify.sh in=reads.fq.gz out=clumped.fa.gz zl=9 pigz fastawrap=100000

This will strip off the names, merge the reads when possible, and then clumpify everything. The output will be fasta-formatted to remove the quality values and have one read per line (unless the reads are over 100kbp long). If quality values need to be saved, then output as “clumped.fq.gz” instead.

  • BBTools User Guide
    • Installation Guide
    • Usage Guide
    • Data Preprocessing
    • Add Adapters Guide
    • BBDuk Guide
    • BBMap Guide
    • BBMask Guide
    • BBMerge Guide
    • BBNorm Guide
    • CalcUniqueness Guide
    • Clumpify Guide
    • Dedupe Guide
    • Reformat Guide
    • Repair Guide
    • Seal Guide
    • Split Nextera Guide
    • Statistics Guide
    • Tadpole Guide
    • Taxonomy Guide
  • BBTools FAQ and Support Forums

More topics:

  • COVID-19 Status
  • News
  • Science Highlights
  • Blog
  • Podcasts
  • CSP Plans
  • Featured Profiles
  • Careers
  • Contact Us
  • Events
  • User Meeting
  • MGM Workshops
  • Internal
  • Disclaimer
  • Credits
  • Emergency Info
  • Accessibility / Section 508 Statement
  • RSS feed
  • Flickr
  • LinkedIn
  • Twitter
  • YouTube
Lawrence Berkeley National Lab Biosciences Area
A project of the US Department of Energy, Office of Science

JGI is a DOE Office of Science User Facility managed by Lawrence Berkeley National Laboratory

© 1997-2021 The Regents of the University of California