BOL: Related items

SpeedSeq

Jit — Fri, 20 Jan 2017 06:05:43 -0600

A flexible framework for rapid genome analysis and interpretation

C Chiang, R M Layer, G G Faust, M R Lindberg, D B Rose, E P Garrison, G T Marth, A R Quinlan, and I M Hall. SpeedSeq: ultra-fast personal genome analysis and interpretation. Nat Meth (2015). doi:10.1038/nmeth.3505.

http://www.nature.com/nmeth/journal/vaop/ncurrent/full/nmeth.3505.html

Address of the bookmark: https://github.com/hall-lab/speedseq

HECIL: A Hybrid Error Correction Algorithm for Long Reads with Iterative Learning

Abhimanyu Singh — Tue, 01 Jan 2019 12:01:00 -0600

HECIL—Hybrid Error Correction with Iterative Learning—a hybrid error correction framework that determines a correction policy for erroneous long reads, based on optimal combinations of decision weights obtained from short read alignments.

HECIL’s core algorithm by introducing an iterative learning paradigm that enhances the correction policy at each iteration by incorporating knowledge gathered from previous iterations via data-driven confidence metrics assigned to prior corrections.

Address of the bookmark: https://github.com/NDBL/HECIL

Common steps for reads mapping !

BioStar — Thu, 09 Mar 2023 02:48:02 -0600

Mapping reads to a reference genome is an essential step in many types of genomic analysis, such as variant calling and gene expression analysis. Here are some general steps to follow for mapping reads to a genome:

Choose a read mapper: There are many read mappers available, such as BWA, Bowtie, and HISAT2. Choose a mapper that is appropriate for your type of data and research question.
Index the reference genome: Before mapping reads, the reference genome needs to be indexed. This involves creating an index of the genome sequence that allows the mapper to quickly find matches to the reads. Most mappers have their own indexing tools.
Prepare the read data: The reads should be in a format that is compatible with the mapper. Most mappers accept FASTQ or BAM files. Depending on the quality of the data, it may need to be filtered or trimmed before mapping.
Run the mapper: The mapper is run with the command-line interface or using a graphical user interface. The specific command depends on the mapper being used, but typically involves specifying the input data, reference genome, and output file format.
Evaluate the mapping results: After the mapping is complete, the results should be evaluated. This includes assessing the quality of the mapping, such as the mapping rate, the number of mapped reads, and the mapping quality score.
Post-processing: Depending on the analysis being performed, post-processing of the mapped reads may be necessary. This can include filtering reads based on quality, removing duplicate reads, and calling variants.

Overall, mapping reads to a reference genome is a complex process that requires careful consideration of the type of data, the research question, and the specific mapper being used.

Simulating the Unseen: New Tools Reshaping Metagenomic Research

LEGE — Wed, 23 Sep 2026 09:42:30 -0500

When benchmarking a new metagenomic pipeline the first step is generally to face a simple problem. A bioinformatician might have a good deal of real sequencing data, but there's a drawback in that the actual biological composition underlying those reads is almost never known. What organisms really were there? In what quantities? How many sequencing errors occurred? Can the pipeline correctly reconstruct the community? Since the true answers are unknown, it is difficult to deal with these questions.

Metagenomic simulators are a way of dealing with this issue; they produce artificial sequencing datasets in which both the composition of the microorganisms and the expected results are known. Researchers can then test a pipeline on such a controlled dataset and see precisely how well it functions. However, as metagenomic studies have become larger, more diverse, and more complex, simply generating artificial reads is no longer sufficient. The simulators now have to replicate community variation, sequencing errors, different sequencing platforms, massive data volumes, and even laboratory-based biases.

MIDASim (2024) was designed with a particular emphasis on microbial community dynamics. Unlike previous tools which regarded each simulated sample as individual, MIDASim is able to efficiently generate realistic multi-sample microbiome datasets, covering the cases where changes over time are being tracked. This feature is particularly useful when investigating how microbial communities vary between people, different experimental setups, or at various times. By overcoming the computational limitations of earlier simulation methods, MIDASim has made it easier to carry out large and complex microbiome simulation studies.

A major challenge is scale. Since modern metagenomic projects can involve millions or even billions of sequencing reads, generating such large synthetic datasets can become a significant computational problem. In order to tackle this issue, Izzy (2025) was developed. As a high-throughput metagenomic read simulator, Izzy is designed with speed in mind while still reproducing the typical patterns of sequencing errors. The people who developed it stated that it can be as much as 60 times faster than earlier tools, which makes large-scale simulations far more practical. Now, researchers can begin to ask not how long a simulation will take but rather what size dataset they want to test.

At the same time, the sequencing technology became more diverse. Researchers were not any longer restricted to using Illumina short reads since Oxford Nanopore and PacBio long-read sequencing also became important in the field of metagenomic research. In order to address these new requirements, MeSS (Metagenomic Sequence Simulator, 2025) was developed. The programme is based on a Snakemake framework and is therefore compatible with the Illumina, Oxford Nanopore and PacBio platforms. MeSS has also been designed to operate efficiently and to use less memory than earlier tools; it provides ready-made templates for the microbiomes of different sites in the human body, thus giving researchers a simple means of beginning to generate realistic simulated communities.

The simulation problem became even more specialized. Suppose that sequencing is not based on uniform sampling of DNA? In the case of targeted sequencing and hybridization-capture experiments, particular sequences are deliberately enriched, and the probability of capturing a target depends on both the probe and the target sequence. A standard model based on uniform sampling might fail to take into account this important feature of real experiments. That is why RAmpSim has been designed specifically to handle simulations for capture-based and targeted sequencing. Instead of depending solely on the assumption of uniform sampling, it includes a thermodynamic nearest-neighbor energy model to represent probe–target interactions and sequence-dependent enrichment. As a result, it is especially suitable for applications such as hybridization capture in metagenomics, where the efficiency of recovering a sequence can depend strongly on how it relates to the capture probes. RAmpSim thus reflects a wider trend towards simulations that aim not only to reproduce sequencing output but also key aspects of the experimental process itself.

MIDASim, Izzy, MeSS and RAmpSim all demonstrate how fast metagenomic simulation is evolving. MIDASim is able to deal with realistic variation within a large number of and changing microbial communities. Izzy overcomes the problem of generating very large datasets. MeSS provides support for a number of sequencing technologies in an efficient and repeatable manner. RAmpSim increases the experimental realism for capture-based sequencing. Although each of these tools addresses a separate issue, they all contribute to advancing the field.

Modern metagenomic simulation therefore aims not only at producing artificial FASTQ files. Researchers nowadays desire simulated datasets that reflect community changes, incorporate the characteristics of the sequencing platform, include real error patterns, take into account computational scale, and account for experimental biases. As metagenomic workflows become more advanced, the simulators have to keep up with this development. The tools are now going beyond that of simple data generators; they produce controlled versions of complex metagenomic experiments and thus provide researchers with something that ordinary sequencing data seldom does: a dataset in which the answer is known before any analysis takes place.

GAM-NGS: genomic assemblies merger for next generation sequencing

Jit — Fri, 19 May 2017 07:44:14 -0500

GAM-NGS is a tool able to merge two or more assemblies in order to improve contiguity and correctness. It can be used on all NGS-based assembly projects and it shows its full potential with multi-library Illumina-based projects. With more than 20 available assemblers it is hard to select the best tool. In this context we propose a tool that improves assemblies (and, as a by-product, perhaps even assemblers) by merging them and selecting the generating that is most likely to be correct.

Address of the bookmark: https://github.com/vice87/gam-ngs

CAMSA :: a tool for Comparative Analysis and Merging of Scaffold Assemblies

Rahul Nayak — Thu, 28 Dec 2017 09:10:26 -0600

CAMSA – is a tool for Comparative Analysis and Merging of Scaffold Assemblies, distributed both as a standalone software package and as Python library under the MIT license.

Main features:

works with any number of scaffold assemblies in de-novo non-progressive fashion
allows to simultaneously work with scaffold assemblies obtained from any in silico and in vitro techniques, supporting multiple existing formats via built-in converters
creates an extensive report with several comparative quality metrics (both on assembly level and on the level of individual assembly points)
constructs a merged combined scaffold assembly
provides an interactive framework for a visual comparative analysis of the given assemblies

Address of the bookmark: https://cblab.org/camsa/

Minipolish: A tool for Racon polishing of miniasm assemblies

BioStar — Tue, 03 Dec 2019 02:40:54 -0600

Miniasm is a great long-read assembly tool: straight-forward, effective and very fast. However, it does not include a polishing step, so its assemblies have a high error rate – they are essentially made of stitched-together pieces of long reads.

Racon is a great polishing tool that can be used to clean up assembly errors. It's also very fast and well suited for long-read data. However, it operates on FASTA files, not the GFA graphs that miniasm makes.

That's where Minipolish comes in. With a single command, it will use Racon to polish up a miniasm assembly, while keeping the assembly in graph form.

It also takes care of some of the other nuances of polishing a miniasm assembly:

Adding read depth information to contigs
Fixing sequence truncation that can occur in Racon
Adding circularising links to circular contigs if not already present (so they display better in Bandage)
'Rotating' circular contigs between polishing rounds to ensure clean circularisation

Address of the bookmark: https://github.com/rrwick/Minipolish

CheckM:Assessing the quality of microbial genomes recovered from isolates, single cells, and metagenomes

Rahul Nayak — Wed, 23 May 2018 04:39:26 -0500

CheckM provides a set of tools for assessing the quality of genomes recovered from isolates, single cells, or metagenomes. It provides robust estimates of genome completeness and contamination by using collocated sets of genes that are ubiquitous and single-copy within a phylogenetic lineage. Assessment of genome quality can also be examined using plots depicting key genomic characteristics (e.g., GC, coding density) which highlight sequences outside the expected distributions of a typical genome. CheckM also provides tools for identifying genome bins that are likely candidates for merging based on marker set compatibility, similarity in genomic characteristics, and proximity within a reference genome tree.

Address of the bookmark: http://ecogenomics.github.io/CheckM/

Public Databases for Bioinformatics !

Jit — Tue, 23 Mar 2021 05:32:15 -0500

https://www.nature.com/articles/s41467-020-17155-y

Server Infrastructure:

File Server:

dhara: Synology 3614 Storage Appliance
4 Core Xeon
108TB disk storage
10Gb ethernet to SCG3
Access atx: dhara:5000
Has btsync server (try it - its much better than dropbox)

Compute Servers:

nandi: Kundaje and Phi Server
24 intel cores
256GB RAM
500GB of SSD storage 
36TB RAID6 local storage
4 Intel Phi's (space for 4 more GPU's)


durga: Montgomery and sensitive data
24 intel cores
256GB RAM
500GB of SSD RAID0 storage 
60TB RAID6 local storage

mitra: Bassik and Web/DB Server
24 core
256GB RAM 
500GB of SSD RAID0 storage 
36TB RAID6 local storage

vayu: Kundaje GPU server
4 core
64GB RAM 
200GB of SSD storage 
8TB RAID10 local storage
4 Nvidia GTX 970 4GB GPUs

amold: Bickel and SGE server
32 AMD core
128GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

wotan: Bickel and SGE server
64 AMD core
256GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

Filesystem:

/users/$USER
default home directory
full backups nightly 
nfs mount to dhara
should store code, papers, and other highly processed data here

/mnt/data/
globally accessible data
should store common data here
e.g. genomes and indexes, annotations, ENCODE data  
if you dont want this to count towards your quote you must chown

/mnt/lab_data/$LAB/
lab accessible data
should store lab project data here 
e.g. ATAC-seq prediction data, enhancer prediction, motif calls

/srv/scratch/$USER
fast local storage
not backed up, but on raid and data will never be deleted
most analysis should be performed here

/srv/persistent/$USER
fast local storage
synced nightly, but not backed up
       ie if the hard drives fail or you delete something and notice 
       within 24 hours we can recover. Otherwise not. (vs home which is 
       properly backed up )  
intermediate analysis products that would be hard to recover should be stored here 
       e.g. stochastic analysis results that need to be kept so that paper 
       results can be reproduced

/srv/www/$LABNAME/
web accessible from mitra.stanford.edu
*NOT BACKED UP*

Some parallel programming patterns:

# gzip a bunch of files
parallel gzip -- *.FILESTOGZIP

# fork example in python:
(for more detailed examples look at 
 https://github.com/nboley/grit/ grit/lib/multiprocessing_utils.py)

import os
import time
import random

import multiprocessing

class ProcessSafeOPStream( object ):
    def __init__( self, writeable_obj ):
        self.writeable_obj = writeable_obj
        self.lock = multiprocessing.Lock()
        self.name = self.writeable_obj.name
        return
    
    def write( self, data ):
        self.lock.acquire()
        self.writeable_obj.write( data )
        self.writeable_obj.flush()
        self.lock.release()
        return
    
    def close( self ):
        self.writeable_obj.close()

def worker(queue, ofp):
    # Try without this
    random.seed()
    while True:
        i = queue.get()
        if i == 'FINISHED': return
        # simulate an expensive function
        x = random.random()
        time.sleep(x/10)
        print i, x
        ofp.write("%i\t%s\n" % (i, x))

NSIMS = 10000
NPROC = 25

# populate queue
todo = multiprocessing.Queue()
for i in xrange(NSIMS): todo.put(i)
for i in xrange(NPROC): todo.put('FINISHED')

ofp = ProcessSafeOPStream( open("output.txt", "w") )

pids = []
for i in xrange(NPROC):
    pid = os.fork()
    if pid == 0:
       worker(todo, ofp)
       os._exit(0)
    else:
       pids.append(pid)  

for pid in pids:
    os.waitpid(pid, 0)

ofp.close()

print "FINISHED"

For use case 1 we obtained the following ENCODE and ROADMAP datasets https://www.encodeproject.org/files/ENCFF446WOD/@@download/ENCFF446WOD.bed.gz, https://www.encodeproject.org/files/ENCFF546PJU/@@download/ENCFF546PJU.bam, https://www.encodeproject.org/files/ENCFF059BEU/@@download/ENCFF059BEU.bam. Blacklisted regions were obtained from http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/hg38-human/hg38.blacklist.bed.gz. The human genome version hg38 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz.

For use case 2 we used the set of narrowPeak files summarized in https://github.com/wkopp/janggu_usecases/tree/master/extra/urls.txt (archived version v1.0.1). The human genome version hg19 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/hg19.fa.gz

For use case 3 we used the ENCODE datasets https://www.encodeproject.org/files/ENCFF591XCX/@@download/ENCFF591XCX.bam, https://www.encodeproject.org/files/ENCFF736LHE/@@download/ENCFF736LHE.bigWig, https://www.encodeproject.org/files/ENCFF177HHM/@@download/ENCFF177HHM.bam as we as the GENCODE annotation v29 from ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_29/gencode.v29.annotation.gtf.gz.

Address of the bookmark: http://mitra.stanford.edu/

Ribbon: Visualizing complex genome alignments and structural variation:

Jit — Wed, 29 Nov 2017 07:40:22 -0600

Ribbon can be used for long reads, short reads, paired-end reads, and assembly/genome alignments. Instructions for each data format are available by clicking on "instructions" in each tab on the right.

Local installation:

You can install Ribbon locally from Github by following the instructions here: https://github.com/MariaNattestad/Ribbon

Address of the bookmark: http://genomeribbon.com/