DOE Joint Genome Institute

  • COVID-19
  • About
  • Phones
  • Contacts
  • Our Science
    • DOE Mission Areas
    • Bioenergy Research Centers
    • Science Programs
    • Products
    • Science Highlights
    • Scientists
    Screencap of green algae video for PNAS paper
    Green Algae Reveal One mRNA Encodes Many Proteins
    A team of researchers has found numerous examples of polycistronic expression – in which two or more genes are encoded on a single molecule of mRNA – in two species of green algae.

    Read more

    Advances in Rapidly Engineering Non-model Bacteria
    CRAGE is a technique for chassis (or strain)-independent recombinase-assisted genome engineering, allowing scientists to conduct genome-wide screens and explore biosynthetic pathways. Now, CRAGE is being applied to other synthetic biology problems.

    Read more

    Maize can produce a cocktail of antibiotics with a handful of enzymes. (Sam Fentress, CC BY-SA 2.0)
    How Maize Makes An Antibiotic Cocktail
    Zealexins are produced in every corn variety and protect maize by fending off fungal and microbial infections using surprisingly few enzymes.

    More

  • Our Projects
    • Search JGI Projects
    • DOE Metrics/Statistics
    • Approved User Proposals
    • Legacy Projects
    Poplar (Populus trichocarpa and P. deltoides) grow in the Advanced Plant Phenotyping Laboratory (APPL) at Oak Ridge National Laboratory in Tennessee. Poplar is an important biofuel feedstock, and Populus trichocarpa is the first tree species to have its genome sequenced — a feat accomplished by JGI. (Image courtesy of Oak Ridge National Laboratory, U.S. Dept. of Energy)
    Podcast: Xiaohan Yang on A Plantiful Future
    Building off plant genomics collaborations between the JGI and Oak Ridge National Laboratory, Xiaohan Yang envisions customizing plants for the benefit of human society.

    More:

    Expansin complex with cell wall in background. (Courtesy of Daniel Cosgrove)
    Synthesizing Microbial Expansins with Unusual Activities
    Expansin proteins from diverse microbes have potential uses in deconstructing lignocellulosic biomass for conversion to renewable biofuels, nanocellulosic fibers, and commodity biochemicals.

    Read more

    High oleic pennycress. (Courtesy of Ratan Chopra)
    Pennycress – A Solution for Global Food Security, Renewable Energy and Ecosystem Benefits
    Pennycress (Thlaspi arvense) is under development as a winter annual oilseed bioenergy crop. It could produce up to 3 billion gallons of seed oil annually while reducing soil erosion and fertilizer runoff.

    Read more

  • Data & Tools
    • IMG
    • Genome Portal
    • MycoCosm
    • PhycoCosm
    • Phytozome
    • GOLD
    Artistic interpretation of CheckV assessing virus genome sequences from environmental samples. (Rendered by Zosia Rostomian​, Berkeley Lab)
    An Automated Tool for Assessing Virus Data Quality
    CheckV can be broadly utilized by the research community to gauge virus data quality and will help researchers to follow best practices and guidelines for providing the minimum amount of information for an uncultivated virus genome.

    More

    Unicellular algae in the Chlorella genus, magnified 1300x. (Andrei Savitsky)
    A One-Stop Shop for Analyzing Algal Genomes
    The PhycoCosm data portal is an interactive browser that allows algal scientists and enthusiasts to look deep into more than 100 algal genomes, compare them, and visualize supporting experimental data.

    More

    Artistic interpretation of how microbial genome sequences from the GEM catalog can help fill in gaps of knowledge about the microbes that play key roles in the Earth's microbiomes. (Rendered by Zosia Rostomian​, Berkeley Lab)
    Podcast: A Primer on Genome Mining
    In Natural Prodcast: the basics of genome mining, and how JGI researchers conducted it in IMG/ABC on thousands of metagenome-derived genomes for a Nature Biotechnology paper.

    Read more

  • User Programs
    • Calls for User Proposals
    • Special Initiatives & Programs
    • User Support
    • Submit a Proposal
    screencap long reads webinar_ Metagenome Program
    Utilizing long-read sequencing for metagenomics and DNA modification detection webinar
    Watch the webinar on how the JGI employs single-molecule, long-read DNA sequences to aid with genome assembly and transcriptome analysis of microbial, fungal, and plant research projects.

    More

    SIP engagement webinar
    “SIP technologies at EMSL and JGI” Webinar
    The concerted stable isotope-related tools and resources of the JGI and the Environmental Molecular Sciences Laboratory (EMSL) may be requested by applying for the annual “Facilities Integrating Collaborations for User Science” (FICUS) call.

    Read more

    martin-adams-unsplash
    CSP Functional Genomics Call Ongoing
    The CSP Functional Genomics call helps users translate genomic information into biological function. Proposals submitted by July 31, 2021 will be part of the next review.

    Read more

  • News & Publications
    • News
    • Blog
    • Podcasts
    • Webinars
    • Publications
    • Newsletter
    • Logos and Templates
    • Photos
    Aerial photo of the switchgrass diversity panel late in the 2020 season at the Kellogg Biological Station in Michigan. (Robert Goodwin)
    A Team Effort Toward Targeted Crop Improvements
    A multi-institutional team has produced a high-quality reference sequence of the complex switchgrass genome. Building off this work, researchers at three DOE Bioenergy Research Centers have expanded the network of common gardens and are exploring improvements to switchgrass.

    More

    Artistic interpretation of how microbial genome sequences from the GEM catalog can help fill in gaps of knowledge about the microbes that play key roles in the Earth's microbiomes. (Rendered by Zosia Rostomian​, Berkeley Lab)
    Uncovering Novel Genomes from Earth’s Microbiomes
    A public repository of 52,515 microbial draft genomes generated from environmental samples around the world, expanding the known diversity of bacteria and archaea by 44%, is now available .

    More

    Green millet (Setaria viridis) plant collected in the wild. (Courtesy of the Kellogg lab)
    Shattering Expectations: Novel Seed Dispersal Gene Found in Green Millet
    In Nature Biotechnology, a very high quality reference Setaria viridis genome was sequenced, and for the first time in wild populations, a gene related to seed dispersal was identified.

    More

Data & Tools
Home › Data & Tools › BBTools › BBTools User Guide › BBMerge Guide

BBMerge Guide

BBMerge is designed to merge two overlapping paired reads into a single read. For example, a 2x150bp read pair with an insert size of 270bp would result in a single 270bp read. This is useful in amplicon studies, as clustering and consensus are far easier with single reads than paired reads, and also in assembly, where longer reads allow the use of longer kmers (for kmer-based assemblers) or fewer comparisons (for overlap-based assemblers). And in either case, the quality of the overlapping bases is improved. BBMerge is also capable of error-correcting the overlapping portion of reads without merging them, as well as merging nonoverlapping reads, if enough coverage is available. BBMerge is the fastest and by far the most accurate overlap-based read merger currently in existence.

BBMerge’s parameters are described in its shell script (bbmerge.sh). This file provides usage examples of various common tasks.

*Notes*

Memory and Kmer Operations:

BBMerge has 2 shell scripts, bbmerge.sh and bbmerge-auto.sh. They are equivalent except for memory usage; so if you override memory auto detection with the -Xmx flag, they are equivalent. bbmerge.sh is designed for overlap-based merging only, and uses a fixed 1GB of RAM (though it can function with much less than that). bbmerge-auto.sh attempts to grab all available physical memory. It is designed for kmer-based operations using Tadpole, which include both merging overlapping and non-overlapping reads, kmer-based error-correction, and kmer-based filtering. If you use an option such as extend, extend2, ecct, kfilter, rem, or rsem, then BBMerge will automatically store kmers. This will use much more time and memory but potentially have various advantages like increased accuracy and increased merge rate of longer-insert pairs. Kmer-based operations should only be used with shotgun (randomly-fragmented) libraries, never with amplicon libraries (such as 16S). They require sufficient coverage; 5x is typically enough, but more is better.

Output streams:

BBMerge supports “out” (aka “outm” or “outmerged”) and “outu” (“outunmerged”). Reads that are merged, or mergeable, go to out, and the rest go to outu. There is a “join” flag (default true) that controls whether mergeable reads get merged. If it is set to false, mergeable reads will be written interleaved to out. All output streams are optional.

Threads and speed:

BBMerge is multithreaded and scales linearly with the number of processor cores, so it’s best to let it automatically use all of them. You can restrict the number of worker threads with the “t” flag if you are working on a shared node. To achieve the maximal speed on a system with many (20+) cores, BBMerge should be fed two files (using in1 and in2) rather than a single interleaved file.
JNI acceleration:

BBMerge has an optional C component (written by Jonathan Rood) which will accelerate merging by approximately 20%. This can be activated with the “jni” flag, but it must first be compiled. For details on compiling it, see /bbmap/jni/README.txt

Strictness:

BBMerge has a lot of settings controlling merging stringency, such as maxratio, ratiomargin, entropy, efilter, etc. Advanced users should feel free to tune these as needed. But it’s a lot simpler to use the predefined strictness levels which adjust the specific settings according to the results of extensive benchmarking. To use a predefined strictness level, simply add a flag like “loose” (you don’t need to add one for “default”). The predefined strictness levels, from strictest to loosest:
xstrict, ustrict, vstrict, strict, default, loose, vloose, uloose, xloose
Stricter settings have lower merge rates and fewer false positives; looser settings have higher merge rates and more false positives. Loose settings are generally, not necessary except with low-quality data (which often happens in low-diversity amplicon sequencing using long reads). A false-positive means a read pair that merged with the wrong overlap length – these can cause problems in assembly or clustering (leading to spurious clusters). However, at any level of strictness, BBMerge has by far the lowest false-positive rate of any read merger, in my testing.

Trimming:

Adapter-trimming reads is fine and is recommended prior to running BBDuk, particularly if kmers will be used. Quality-trimming is usually not recommended, unless the library has major quality issues at the right end of reads resulting in a low merge rate, in which case only weak trimming (such as qtrim=r trimq=8) should be used.

When not to use:

If you run BBMerge, and under, say, 15% of the reads merge, even at very loose stringency, it’s probably a waste of time to merge – you’ll just make the workflow more complicated, and possibly get a lot of false-positives. Also, don’t try to merge single-ended libraries or long-mate-pair libraries that are not in an “innie” orientation. Generally you should not be merging LMP libraries at all except for special analysis purposes, such as determining what fraction of your LMP library is actually short-insert fragments.

*Usage Examples*

Basic merging:

bbmerge.sh in=reads.fq out=merged.fq outu=unmerged.fq ihist=ihist.txt

This will merge the reads by overlap. If no best overlap is found, the pair will go to outu; otherwise, the reads will be merged and sent to out. After finishing, an insert size histogram will be written to ihist.txt. This can be produced even if “out” or “outu” are not specified.

Overlap-based error-correction:

bbmerge.sh in=reads.fq out=corrected.fq ecco mix

This will correct reads that overlap, rather than merging them. Where the two reads agree, the quality score will be increased; where they disagree, the score will be reduced, and the base call will be changed to the base with the higher quality. If the bases differ and the scores are equal, the base will be replaced with N.

Merging of nonoverlapping reads using kmers:

bbmerge-auto.sh in=reads.fq out=merged.fq outu=unmerged.fq ihist=ihist.txt ecct extend2=20 iterations=5

This will attempt to merge each pair by overlap. If unsuccessful, both reads will be error-corrected using Tadpole, and then merging will be tried again. If still unsuccessful, both reads will be extended by 20bp, then merging will be attempted again. This will repeat up to 5 times, or until neither of the reads can be extended any more due to a branch or dead-end in the kmer graph. If the reads are not merged, all of the changes are undone and the original pair will be sent to outu. “extend2=20 iterations=5” will extend each read by up to 100bp, which increases the maximum insert sizes that can be merged by 200bp. So, for example, a 2x150bp library can normally only merge inserts up to around 290bp; this would extend that capability to 490bp, and the middle would be filled in with assembled bases.

Using kmers to reduce false positives:

bbmerge-auto.sh in=reads.fq out=merged.fq outu=unmerged.fq rem extend2=50 k=62
or
bbmerge-auto.sh in=reads.fq out=merged.fq outu=unmerged.fq rsem extend2=50 k=62

This is similar to the above section on merging nonoverlapping reads, but the goal is to perform conservative merges, particularly in repetitive areas. “rem” or “requireextensionmatch” will try to merge the raw reads, then try to extend them, then merge them again. If the two merges give the same result, then the reads will be merged. If they give different results, the reads will only be merged if the raw reads gave no solution but the extended reads gave a solution indicating the raw reads don’t overlap, or if the raw reads DID have a solution and no extension was possible. “rsem” is stricter, in that if the raw reads did not give a solution, extension will not be attempted. In practice rsem has a lower false-positive merge rate than rem, but rem allows non-overlapping merges.

Discovering adapter sequences:

bbmerge.sh in=reads.fq outa=adapters.fa

This will report the consensus adapter sequences of pairs with insert size shorter than read length. The adapter sequences can then be used for trimming with BBDuk or fed back into BBMerge to improve merging accuracy.
Using adapter sequences to improve merging accuracy:
bbmerge.sh in=reads.fq out=merged.fq adapter1=GATCGGAAGAGCACACGTCTGAACTCCAGTC adapter2=GATCGGAAGAGCACACGTCTGAACTCCAGTC
or
bbmerge.sh in=reads.fq out=merged.fq adapters=adapters.fa

The argument for adapter1=, adapter2=, or adapters= can be a literal sequence, a fasta file, or a comma-delimited list of sequences and/or fasta files. The adapter sequences will only be used to ensure that reads that overlap with an implied insert size of less than read length actually contain adapter sequence at the expected location. This is optional but can substantially increase accuracy, so it is highly recommended if you know the adapter sequence. Do not use this option if the left end of reads have been trimmed in any way. Trimming the right end is fine; e.g., ktrim=r or qtrim=r in BBDuk. However, it is generally not recommended to do any quality-trimming prior to merging.

Allowing perfect overlaps only:

bbmerge.sh in=reads.fq out=merged.fq pfilter=1

“pfilter” bans merges in which the probability of the resulting mismatches falls below a specified value, based on the quality values of the bases that don’t match. “pfilter=1” is the strictest possible setting, which bans merges in which there are any mismatches in the overlap region.

Recommended command for optimal accuracy:

bbmerge-auto.sh in=reads.fq out=merged.fq adapter1=something adapter2=something rem k=62 extend2=50 ecct

If you have sufficient depth for kmer-based extension and error-correction (and are using a shotgun fragment library), and you know the adapter sequences, this command will maximize the correct merges and minimize the incorrect merges, in my testing. You can also add a strictness modifier (such as “loose” or “vstrict”) as desired. I typically use vstrict when preparing reads for assembly and loose when calculating an insert size distribution.

  • BBTools User Guide
    • Installation Guide
    • Usage Guide
    • Data Preprocessing
    • Add Adapters Guide
    • BBDuk Guide
    • BBMap Guide
    • BBMask Guide
    • BBMerge Guide
    • BBNorm Guide
    • CalcUniqueness Guide
    • Clumpify Guide
    • Dedupe Guide
    • Reformat Guide
    • Repair Guide
    • Seal Guide
    • Split Nextera Guide
    • Statistics Guide
    • Tadpole Guide
    • Taxonomy Guide
  • BBTools FAQ and Support Forums

More topics:

  • COVID-19 Status
  • News
  • Science Highlights
  • Blog
  • Podcasts
  • Webinars
  • CSP Plans
  • Featured Profiles
  • Careers
  • Contact Us
  • Events
  • User Meeting
  • MGM Workshops
  • Internal
  • Disclaimer
  • Credits
  • Emergency Info
  • Accessibility / Section 508 Statement
  • RSS feed
  • Flickr
  • LinkedIn
  • Twitter
  • YouTube
Lawrence Berkeley National Lab Biosciences Area
A project of the US Department of Energy, Office of Science

JGI is a DOE Office of Science User Facility managed by Lawrence Berkeley National Laboratory

© 1997-2021 The Regents of the University of California