BOL: Related items

The DNA of a Successful Bioinformatician decoded !!!

Jit — Wed, 12 Mar 2014 13:41:26 -0500

Many blogs exist about successful bioinformatician, but this blog so far now is my personal view on characteristics of successful bioinformatician or computational biologist. Hmm … of course these views are subjective to my own personal experiences and therefore I don't claim that the view listed here is complete. As a human, I don’t take them too serious. The success must not be the only target of your work. The target is to work on your own virtues; some of those virtues are the topic of this blog.

1. Update new things continuously
As per my personal experience, it’s not always easy to work as a bioinformatician! There are couple of reasons to say that; First computational part of biology make our life’s a little harder compared to other professional categories. The fact - for instance - that the technology cycle in the bioinformatics world is very short, the actual knowledge becomes outdated in a few months or years. Therefore, we need to learn continuously - new things get important. Second, to stay on top of things we really need the strong will to be good at our job. That's probably the most important characteristic to bioinformatician. They are usually an excellent knowledge worker with great technical abilities, and have the will to be that over decades!

2. Avoid the sentence "I did not know what to do!"
In our computational biology lab, we generally face lots of technical problems. But as you know, it's impossible to know everything to do the computational biology jobs ( Yup.. because you need diverse and multidisciplinary knowledge to understand biological problems and resolve their respective solutions), therefore it's absolutely necessary that a bioinformatician finds its way through a new topic. How I typically do that is I use google and I talk to other experts in our laboratory or online biostar community to find out what they think. "I did not know what to do!" should not be an argument for us.

3. To make oneself useful
Several time it does happen, you finished our task earlier than expected; in such cases if you have some time left then: Take a coffee and play chess; reversi, etc. In my case I take a rest. Afterwards I think about what I could do that helps the team to achieve its targets, 'cause some of my team mates probably didn't finish! (at least if I didn't met them at coffee bar !!)

4. Care for all
During my rigorous research duration; I attended several workshop organized by my University departments. I had a discussion with other research fellow, professors; I generally ask … what it really takes to make a team successful or to be a successful research leader. They always said: "Well, you need some caring people!" I think there is a lot truth in that statement. If we do not care about quality, timelines, good team culture, respectful communication (!!), clean code, if all this doesn’t matter to us, then I believe the probability is higher that we fail in research and analysis.

5. Be good with people
Because bioinformatician and computational biologist jobs typically involves to work in a (most wanted J cross-departmental!) team, therefore it's important that we're (more or less) good in dealing with other individuals. Everyone have their own strengths and weaknesses, just like us. It's important to treat all the research team mates with respect, regardless of their technical competence or contributions. Of course, sometimes people deserve a clear statement (!!!), but try to do these things one-on-one. Make sure nobody loses his face. Attend the meetings at the coffee bar; be good at table top soccer and go out once in a while to have a beer with your team. You know what I'm talking about.

At the end of a week I look back and I ask myself what I have produced. This could be paperwork, community days or (best!!) programming code. Always remember there is always a solution to a problem. Most of the times there are at least three solutions. So, don’t just blame, suggest a solution.

That's it. I am looking forward to your thoughts and comments!

GAM-NGS: genomic assemblies merger for next generation sequencing

Jit — Fri, 19 May 2017 07:44:14 -0500

GAM-NGS is a tool able to merge two or more assemblies in order to improve contiguity and correctness. It can be used on all NGS-based assembly projects and it shows its full potential with multi-library Illumina-based projects. With more than 20 available assemblers it is hard to select the best tool. In this context we propose a tool that improves assemblies (and, as a by-product, perhaps even assemblers) by merging them and selecting the generating that is most likely to be correct.

Address of the bookmark: https://github.com/vice87/gam-ngs

CAMSA :: a tool for Comparative Analysis and Merging of Scaffold Assemblies

Rahul Nayak — Thu, 28 Dec 2017 09:10:26 -0600

CAMSA – is a tool for Comparative Analysis and Merging of Scaffold Assemblies, distributed both as a standalone software package and as Python library under the MIT license.

Main features:

works with any number of scaffold assemblies in de-novo non-progressive fashion
allows to simultaneously work with scaffold assemblies obtained from any in silico and in vitro techniques, supporting multiple existing formats via built-in converters
creates an extensive report with several comparative quality metrics (both on assembly level and on the level of individual assembly points)
constructs a merged combined scaffold assembly
provides an interactive framework for a visual comparative analysis of the given assemblies

Address of the bookmark: https://cblab.org/camsa/

AccessSyRI: finding genomic rearrangements and local sequence differences from whole-genome assemblies

Jit — Sat, 01 Feb 2020 13:38:49 -0600

AccessSyRI: finding genomic rearrangements andlocal sequence differences from whole-genome assemblies

SyRI, a pairwise whole-genome comparison tool for chromosome-level assemblies. SyRI starts by finding rearranged regions and then searches for differences in the sequences, which are distinguished for residing in syntenic or rearranged regions. This distinction is important as rearranged regions are inherited differently compared to syntenic regions.

https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1911-0

Address of the bookmark: https://github.com/schneebergerlab/syri

shovill: Assemble bacterial isolate genomes from Illumina paired-end reads

BioStar — Sat, 02 Jan 2021 07:05:36 -0600

Shovill is a pipeline which uses SPAdes at its core, but alters the steps before and after the primary assembly step to get similar results in less time. Shovill also supports other assemblers like SKESA, Velvet and Megahit, so you can take advantage of the pre- and post-processing the Shovill provides with those too.

Address of the bookmark: https://github.com/tseemann/shovill

Bactopia: a flexible pipeline for complete analysis of bacterial genomes

Abhi — Sat, 08 Jun 2024 16:25:08 -0500

Bactopia is a flexible pipeline for complete analysis of bacterial genomes. The goal of Bactopia is process your data with a broad set of tools, so that you can get to the fun part of analyses quicker!

Bactopia was inspired by Staphopia, a workflow we (Tim Read and myself) released that is targeted towards Staphylococcus aureus genomes. Using what we learned from Staphopia and user feedback, Bactopia was developed from scratch with usability, portability, and speed in mind from the start.

Bactopia uses Nextflow to manage the workflow, allowing for support of many types of environments (e.g. cluster or cloud). Bactopia allows for the usage of many public datasets as well as your own datasets to further enhance the analysis of your sequencing. Bactopia only uses software packages available from Bioconda and Conda-Forge to make installation as simple as possible for all users.

To highlight the use of Bactopia and Bactopia Tools, we performed an analysis of 1,664 public Lactobacillus genomes, focusing on Lactobacillus crispatus, a species that is a common part of the human vaginal microbiome. The results from this analysis are published in mSystems under the title: Bactopia: a flexible pipeline for complete analysis of bacterial genomes

Address of the bookmark: https://bactopia.github.io/latest/

Bactopia: a Flexible Pipeline for Complete Analysis of Bacterial Genomes

Abhi — Wed, 15 May 2024 14:36:12 -0500

Bactopia is a flexible pipeline for complete analysis of bacterial genomes. The goal of Bactopia is to process your data with a broad set of tools, so that you can get to the fun part of analyses quicker!

Bactopia can be split into two main parts: Bactopia Analysis Pipeline, and Bactopia Tools.

Bactopia Analysis Pipeline is the main per-isolate workflow in Bactopia. Built with Nextflow, input FASTQs (local or available from SRA/ENA) are put through numerous analyses including: quality control, assembly, annotation, minmer sketch queries, sequence typing, and more.

Bactopia Tools are a set a independent workflows fo

Address of the bookmark: https://github.com/bactopia/bactopia

Public Databases for Bioinformatics !

Jit — Tue, 23 Mar 2021 05:32:15 -0500

https://www.nature.com/articles/s41467-020-17155-y

Server Infrastructure:

File Server:

dhara: Synology 3614 Storage Appliance
4 Core Xeon
108TB disk storage
10Gb ethernet to SCG3
Access atx: dhara:5000
Has btsync server (try it - its much better than dropbox)

Compute Servers:

nandi: Kundaje and Phi Server
24 intel cores
256GB RAM
500GB of SSD storage 
36TB RAID6 local storage
4 Intel Phi's (space for 4 more GPU's)


durga: Montgomery and sensitive data
24 intel cores
256GB RAM
500GB of SSD RAID0 storage 
60TB RAID6 local storage

mitra: Bassik and Web/DB Server
24 core
256GB RAM 
500GB of SSD RAID0 storage 
36TB RAID6 local storage

vayu: Kundaje GPU server
4 core
64GB RAM 
200GB of SSD storage 
8TB RAID10 local storage
4 Nvidia GTX 970 4GB GPUs

amold: Bickel and SGE server
32 AMD core
128GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

wotan: Bickel and SGE server
64 AMD core
256GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

Filesystem:

/users/$USER
default home directory
full backups nightly 
nfs mount to dhara
should store code, papers, and other highly processed data here

/mnt/data/
globally accessible data
should store common data here
e.g. genomes and indexes, annotations, ENCODE data  
if you dont want this to count towards your quote you must chown

/mnt/lab_data/$LAB/
lab accessible data
should store lab project data here 
e.g. ATAC-seq prediction data, enhancer prediction, motif calls

/srv/scratch/$USER
fast local storage
not backed up, but on raid and data will never be deleted
most analysis should be performed here

/srv/persistent/$USER
fast local storage
synced nightly, but not backed up
       ie if the hard drives fail or you delete something and notice 
       within 24 hours we can recover. Otherwise not. (vs home which is 
       properly backed up )  
intermediate analysis products that would be hard to recover should be stored here 
       e.g. stochastic analysis results that need to be kept so that paper 
       results can be reproduced

/srv/www/$LABNAME/
web accessible from mitra.stanford.edu
*NOT BACKED UP*

Some parallel programming patterns:

# gzip a bunch of files
parallel gzip -- *.FILESTOGZIP

# fork example in python:
(for more detailed examples look at 
 https://github.com/nboley/grit/ grit/lib/multiprocessing_utils.py)

import os
import time
import random

import multiprocessing

class ProcessSafeOPStream( object ):
    def __init__( self, writeable_obj ):
        self.writeable_obj = writeable_obj
        self.lock = multiprocessing.Lock()
        self.name = self.writeable_obj.name
        return
    
    def write( self, data ):
        self.lock.acquire()
        self.writeable_obj.write( data )
        self.writeable_obj.flush()
        self.lock.release()
        return
    
    def close( self ):
        self.writeable_obj.close()

def worker(queue, ofp):
    # Try without this
    random.seed()
    while True:
        i = queue.get()
        if i == 'FINISHED': return
        # simulate an expensive function
        x = random.random()
        time.sleep(x/10)
        print i, x
        ofp.write("%i\t%s\n" % (i, x))

NSIMS = 10000
NPROC = 25

# populate queue
todo = multiprocessing.Queue()
for i in xrange(NSIMS): todo.put(i)
for i in xrange(NPROC): todo.put('FINISHED')

ofp = ProcessSafeOPStream( open("output.txt", "w") )

pids = []
for i in xrange(NPROC):
    pid = os.fork()
    if pid == 0:
       worker(todo, ofp)
       os._exit(0)
    else:
       pids.append(pid)  

for pid in pids:
    os.waitpid(pid, 0)

ofp.close()

print "FINISHED"

For use case 1 we obtained the following ENCODE and ROADMAP datasets https://www.encodeproject.org/files/ENCFF446WOD/@@download/ENCFF446WOD.bed.gz, https://www.encodeproject.org/files/ENCFF546PJU/@@download/ENCFF546PJU.bam, https://www.encodeproject.org/files/ENCFF059BEU/@@download/ENCFF059BEU.bam. Blacklisted regions were obtained from http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/hg38-human/hg38.blacklist.bed.gz. The human genome version hg38 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz.

For use case 2 we used the set of narrowPeak files summarized in https://github.com/wkopp/janggu_usecases/tree/master/extra/urls.txt (archived version v1.0.1). The human genome version hg19 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/hg19.fa.gz

For use case 3 we used the ENCODE datasets https://www.encodeproject.org/files/ENCFF591XCX/@@download/ENCFF591XCX.bam, https://www.encodeproject.org/files/ENCFF736LHE/@@download/ENCFF736LHE.bigWig, https://www.encodeproject.org/files/ENCFF177HHM/@@download/ENCFF177HHM.bam as we as the GENCODE annotation v29 from ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_29/gencode.v29.annotation.gtf.gz.

Address of the bookmark: http://mitra.stanford.edu/

Ribbon: Visualizing complex genome alignments and structural variation:

Jit — Wed, 29 Nov 2017 07:40:22 -0600

Ribbon can be used for long reads, short reads, paired-end reads, and assembly/genome alignments. Instructions for each data format are available by clicking on "instructions" in each tab on the right.

Local installation:

You can install Ribbon locally from Github by following the instructions here: https://github.com/MariaNattestad/Ribbon

Address of the bookmark: http://genomeribbon.com/

kSNP3.0: SNP detection and phylogenetic analysis of genomes without genome alignment or reference genome

Jit — Fri, 08 Dec 2017 16:48:40 -0600

Sept. 20, 2017 Version 3.1 released. Major upgrade. Version 3.1 fixes the problems with SNP annotation that arose when NCBI discontinued use of GI numbers. Please read carefully the Preface (page 3) and the File of annotated genomes section (pages 9-10) in the version 3.1 User Guide. Thanks to Tom Slezak for revsing the get_genbank_file3 script and to Tod Stuber (USDA) for testing version 3.1 even though he doesn't need the annotation feature. All users are encouraged to upgrade to version 3.1.

Address of the bookmark: https://sourceforge.net/projects/ksnp/files/