DOE Joint Genome Institute

  • COVID-19
  • About Us
  • Contact Us
  • Our Science
    • DOE Mission Areas
    • Science Programs
    • Science Highlights
    • Scientists
    A vertical tree stump outdoors with about a dozen shiitake mushrooms sprouting from its surface.
    Tracing the Evolution of Shiitake Mushrooms
    Understanding Lentinula genomes and their evolution could provide strategies for converting plant waste into sugars for biofuel production. Additionally, these fungi play a role in the global carbon cycle.

    More

    Soil Virus Offers Insight into Maintaining Microorganisms
    Through a collaborative effort, researchers have identified a protein in soil viruses that may promote soil health.

    More

    Data yielded from RIViT-seq increased the number of sigma factor-gene pairs confirmed in Streptomyces coelicolor from 209 to 399. Here, grey arrows denote previously known regulation and red arrows are regulation identified by RIViT-seq; orange nodes mark sigma factors while gray nodes mark other genes. (Otani, H., Mouncey, N.J. Nat Commun 13, 3502 (2022). https://doi.org/10.1038/s41467-022-31191-w)
    Streamlining Regulon Identification in Bacteria
    Regulons are a group of genes that can be turned on or off by the same regulatory protein. RIViT-seq technology could speed up associating transcription factors with their target genes.

    More

  • Our Projects
    • Search JGI Projects
    • DOE Metrics/Statistics
    • Approved User Proposals
    • Legacy Projects
    A panoramic view of a lake reflecting a granite mountain.
    Genome Insider: Methane Makers in Yosemite’s Lakes
    Meet researchers who sampled the microbial communities living in the mountaintop lakes of the Sierra Nevada mountains to see how climate change affects freshwater ecosystems, and how those ecosystems work.

    Listen

    A light green shrub with spiny leaves, up close.
    Genome Insider: A Shrubbier Version of Rubber
    Hear from the consortium working on understanding the guayule plant's genome, which could lead to an improved natural rubber plant.

    Listen

    The switchgrass diversity panel growing at the Kellogg Biological Station in Michigan. (David Lowry)
    Mapping Switchgrass Traits with Common Gardens
    The combination of field data and genetic information has allowed researchers to associate climate adaptations with switchgrass biology.

    More

  • Data & Tools
    • IMG
    • Data Portal
    • MycoCosm
    • PhycoCosm
    • Phytozome
    • GOLD
    iPHoP image (Simon Roux)
    iPHoP: A Matchmaker for Phages and their Hosts
    Building on existing virus-host prediction approaches, a new tool combines and evaluates multiple predictions to reliably match viruses with their archaea and bacteria hosts.

    More

    Abstract image of gold lights and squares against a black backdrop
    Silver Age of GOLD Introduces New Features
    The Genomes OnLine Database makes curated microbiome metadata that follows community standards freely available and enables large-scale comparative genomics analysis initiatives.

    More

    Graphical overview of the RNA Virus MetaTranscriptomes Project. (Courtesy of Simon Roux)
    A Better Way to Find RNA Virus Needles in the Proverbial Database Haystacks
    Researchers combed through more than 5,000 data sets of RNA sequences generated from diverse environmental samples around the world, resulting in a five-fold increase of RNA virus diversity.

    More

  • User Programs
    • Calls for Proposals
    • Special Initiatives & Programs
    • Product Offerings
    • User Support
    • Policies
    • Submit a Proposal
    Green plant matter grows from the top, with the area just beneath the surface also visible as soil, root systems and a fuzzy white substance surrounding them.
    Supercharging SIP in the Fungal Hyphosphere
    Applying high-throughput stable isotope probing to the study of a particular fungi, researchers identified novel interactions between bacteria and the fungi.

    More

    Digital ID card with six headshots reads: Congratulations to our 2022 Function Genomics recipients!
    Final Round of 2022 CSP Functional Genomics Awardees
    Meet the final six researchers whose proposals were selected for the 2022 Community Science Program Functional Genomics call.

    More

    croppe image of the JGI helix sculpture
    Tips for a Winning Community Science Program Proposal
    In the Genome Insider podcast, tips to successfully avail of the JGI's proposal calls, many through the Community Science Program.

    Listen

  • News & Publications
    • News
    • Blog
    • Podcasts
    • Webinars
    • Publications
    • Newsletter
    • Logos and Templates
    • Photos
    2022 JGI-UC Merced interns (Thor Swift/Berkeley Lab)
    Exploring Possibilities: 2022 JGI-UC Merced Interns
    The 2022 UC Merced intern cohort share how their summer internship experiences have influenced their careers in science.

    More

    image from gif that shows where in the globe JGI fungal collaborators are located.
    Using Team Science to Build Communities Around Data
    As the data portals grow and evolve, the research communities further expand around them. But with two projects, communities are forming to generate high quality genomes to benefit researchers.

    More

    Cow Rumen and the Early Days of Metagenomics
    Tracing a cow rumen dataset from the lab to material for a hands-on undergraduate research course at CSU-San Marcos that has since expanded into three other universities.

    More

Data & Tools
Home › Data & Tools › Software › BBTools › BBTools User Guide › BBMerge Guide

BBMerge Guide

BBMerge is designed to merge two overlapping paired reads into a single read. For example, a 2x150bp read pair with an insert size of 270bp would result in a single 270bp read. This is useful in amplicon studies, as clustering and consensus are far easier with single reads than paired reads, and also in assembly, where longer reads allow the use of longer kmers (for kmer-based assemblers) or fewer comparisons (for overlap-based assemblers). And in either case, the quality of the overlapping bases is improved. BBMerge is also capable of error-correcting the overlapping portion of reads without merging them, as well as merging nonoverlapping reads, if enough coverage is available. BBMerge is the fastest and by far the most accurate overlap-based read merger currently in existence.

BBMerge’s parameters are described in its shell script (bbmerge.sh). This file provides usage examples of various common tasks.

*Notes*

Memory and Kmer Operations:

BBMerge has 2 shell scripts, bbmerge.sh and bbmerge-auto.sh. They are equivalent except for memory usage; so if you override memory auto detection with the -Xmx flag, they are equivalent. bbmerge.sh is designed for overlap-based merging only, and uses a fixed 1GB of RAM (though it can function with much less than that). bbmerge-auto.sh attempts to grab all available physical memory. It is designed for kmer-based operations using Tadpole, which include both merging overlapping and non-overlapping reads, kmer-based error-correction, and kmer-based filtering. If you use an option such as extend, extend2, ecct, kfilter, rem, or rsem, then BBMerge will automatically store kmers. This will use much more time and memory but potentially have various advantages like increased accuracy and increased merge rate of longer-insert pairs. Kmer-based operations should only be used with shotgun (randomly-fragmented) libraries, never with amplicon libraries (such as 16S). They require sufficient coverage; 5x is typically enough, but more is better.

Output streams:

BBMerge supports “out” (aka “outm” or “outmerged”) and “outu” (“outunmerged”). Reads that are merged, or mergeable, go to out, and the rest go to outu. There is a “join” flag (default true) that controls whether mergeable reads get merged. If it is set to false, mergeable reads will be written interleaved to out. All output streams are optional.

Threads and speed:

BBMerge is multithreaded and scales linearly with the number of processor cores, so it’s best to let it automatically use all of them. You can restrict the number of worker threads with the “t” flag if you are working on a shared node. To achieve the maximal speed on a system with many (20+) cores, BBMerge should be fed two files (using in1 and in2) rather than a single interleaved file.
JNI acceleration:

BBMerge has an optional C component (written by Jonathan Rood) which will accelerate merging by approximately 20%. This can be activated with the “jni” flag, but it must first be compiled. For details on compiling it, see /bbmap/jni/README.txt

Strictness:

BBMerge has a lot of settings controlling merging stringency, such as maxratio, ratiomargin, entropy, efilter, etc. Advanced users should feel free to tune these as needed. But it’s a lot simpler to use the predefined strictness levels which adjust the specific settings according to the results of extensive benchmarking. To use a predefined strictness level, simply add a flag like “loose” (you don’t need to add one for “default”). The predefined strictness levels, from strictest to loosest:
xstrict, ustrict, vstrict, strict, default, loose, vloose, uloose, xloose
Stricter settings have lower merge rates and fewer false positives; looser settings have higher merge rates and more false positives. Loose settings are generally, not necessary except with low-quality data (which often happens in low-diversity amplicon sequencing using long reads). A false-positive means a read pair that merged with the wrong overlap length – these can cause problems in assembly or clustering (leading to spurious clusters). However, at any level of strictness, BBMerge has by far the lowest false-positive rate of any read merger, in my testing.

Trimming:

Adapter-trimming reads is fine and is recommended prior to running BBDuk, particularly if kmers will be used. Quality-trimming is usually not recommended, unless the library has major quality issues at the right end of reads resulting in a low merge rate, in which case only weak trimming (such as qtrim=r trimq=8) should be used.

When not to use:

If you run BBMerge, and under, say, 15% of the reads merge, even at very loose stringency, it’s probably a waste of time to merge – you’ll just make the workflow more complicated, and possibly get a lot of false-positives. Also, don’t try to merge single-ended libraries or long-mate-pair libraries that are not in an “innie” orientation. Generally you should not be merging LMP libraries at all except for special analysis purposes, such as determining what fraction of your LMP library is actually short-insert fragments.

*Usage Examples*

Basic merging:

bbmerge.sh in=reads.fq out=merged.fq outu=unmerged.fq ihist=ihist.txt

This will merge the reads by overlap. If no best overlap is found, the pair will go to outu; otherwise, the reads will be merged and sent to out. After finishing, an insert size histogram will be written to ihist.txt. This can be produced even if “out” or “outu” are not specified.

Overlap-based error-correction:

bbmerge.sh in=reads.fq out=corrected.fq ecco mix

This will correct reads that overlap, rather than merging them. Where the two reads agree, the quality score will be increased; where they disagree, the score will be reduced, and the base call will be changed to the base with the higher quality. If the bases differ and the scores are equal, the base will be replaced with N.

Merging of nonoverlapping reads using kmers:

bbmerge-auto.sh in=reads.fq out=merged.fq outu=unmerged.fq ihist=ihist.txt ecct extend2=20 iterations=5

This will attempt to merge each pair by overlap. If unsuccessful, both reads will be error-corrected using Tadpole, and then merging will be tried again. If still unsuccessful, both reads will be extended by 20bp, then merging will be attempted again. This will repeat up to 5 times, or until neither of the reads can be extended any more due to a branch or dead-end in the kmer graph. If the reads are not merged, all of the changes are undone and the original pair will be sent to outu. “extend2=20 iterations=5” will extend each read by up to 100bp, which increases the maximum insert sizes that can be merged by 200bp. So, for example, a 2x150bp library can normally only merge inserts up to around 290bp; this would extend that capability to 490bp, and the middle would be filled in with assembled bases.

Using kmers to reduce false positives:

bbmerge-auto.sh in=reads.fq out=merged.fq outu=unmerged.fq rem extend2=50 k=62
or
bbmerge-auto.sh in=reads.fq out=merged.fq outu=unmerged.fq rsem extend2=50 k=62

This is similar to the above section on merging nonoverlapping reads, but the goal is to perform conservative merges, particularly in repetitive areas. “rem” or “requireextensionmatch” will try to merge the raw reads, then try to extend them, then merge them again. If the two merges give the same result, then the reads will be merged. If they give different results, the reads will only be merged if the raw reads gave no solution but the extended reads gave a solution indicating the raw reads don’t overlap, or if the raw reads DID have a solution and no extension was possible. “rsem” is stricter, in that if the raw reads did not give a solution, extension will not be attempted. In practice rsem has a lower false-positive merge rate than rem, but rem allows non-overlapping merges.

Discovering adapter sequences:

bbmerge.sh in=reads.fq outa=adapters.fa

This will report the consensus adapter sequences of pairs with insert size shorter than read length. The adapter sequences can then be used for trimming with BBDuk or fed back into BBMerge to improve merging accuracy.
Using adapter sequences to improve merging accuracy:
bbmerge.sh in=reads.fq out=merged.fq adapter1=GATCGGAAGAGCACACGTCTGAACTCCAGTC adapter2=GATCGGAAGAGCACACGTCTGAACTCCAGTC
or
bbmerge.sh in=reads.fq out=merged.fq adapters=adapters.fa

The argument for adapter1=, adapter2=, or adapters= can be a literal sequence, a fasta file, or a comma-delimited list of sequences and/or fasta files. The adapter sequences will only be used to ensure that reads that overlap with an implied insert size of less than read length actually contain adapter sequence at the expected location. This is optional but can substantially increase accuracy, so it is highly recommended if you know the adapter sequence. Do not use this option if the left end of reads have been trimmed in any way. Trimming the right end is fine; e.g., ktrim=r or qtrim=r in BBDuk. However, it is generally not recommended to do any quality-trimming prior to merging.

Allowing perfect overlaps only:

bbmerge.sh in=reads.fq out=merged.fq pfilter=1

“pfilter” bans merges in which the probability of the resulting mismatches falls below a specified value, based on the quality values of the bases that don’t match. “pfilter=1” is the strictest possible setting, which bans merges in which there are any mismatches in the overlap region.

Recommended command for optimal accuracy:

bbmerge-auto.sh in=reads.fq out=merged.fq adapter1=something adapter2=something rem k=62 extend2=50 ecct

If you have sufficient depth for kmer-based extension and error-correction (and are using a shotgun fragment library), and you know the adapter sequences, this command will maximize the correct merges and minimize the incorrect merges, in my testing. You can also add a strictness modifier (such as “loose” or “vstrict”) as desired. I typically use vstrict when preparing reads for assembly and loose when calculating an insert size distribution.

  • BBTools User Guide
    • Installation Guide
    • Usage Guide
    • Data Preprocessing
    • Add Adapters Guide
    • BBDuk Guide
    • BBMap Guide
    • BBMask Guide
    • BBMerge Guide
    • BBNorm Guide
    • CalcUniqueness Guide
    • Clumpify Guide
    • Dedupe Guide
    • Reformat Guide
    • Repair Guide
    • Seal Guide
    • Split Nextera Guide
    • Statistics Guide
    • Tadpole Guide
    • Taxonomy Guide
  • BBTools FAQ and Support Forums

More topics:

  • COVID-19 Status
  • News
  • Science Highlights
  • Blog
  • Webinars
  • CSP Plans
  • Featured Profiles
  • Careers
  • Contact Us
  • Events
  • User Meeting
  • MGM Workshops
  • Internal
  • Disclaimer
  • Credits
  • Policies
  • Emergency Info
  • Accessibility / Section 508 Statement
  • Flickr
  • LinkedIn
  • RSS
  • Twitter
  • YouTube
Lawrence Berkeley National Lab Biosciences Area
A project of the US Department of Energy, Office of Science

JGI is a DOE Office of Science User Facility managed by Lawrence Berkeley National Laboratory

© 1997-2023 The Regents of the University of California