BOL: Related items

Linux for bioinformatician !!!

Rahul Nayak — Thu, 13 Mar 2014 16:59:26 -0500

Linux, free operating system for computers, provides several powerful admin tools and utilities which will help you to manage your systems effectively and handle huge amount of genomic/biological data with an ease. The field of bioinformatics relies heavily on Linux-based computers and software. Although most bioinformatics programs can be compiled to run. If you don’t know what these no so user-friendly tools are and how to use them, you could be spending lot of time trying to perform even the basic admin tasks. The focus of this linux series is to help you understand system admin as well as basic tools, which will help you to become an effective bioinformatician and computational biologist.

For knowledge about Linux and their importance amongst bioinformatician plesae read this article "An introduction to Linux for bioinformatics" by Paul Stothard.

Linux cheat sheet at http://bioinformaticsonline.com/file/view/87/linux-cheat-sheet

Please browse for futher useful linux pages on right hand side ...

ExRec: Exclusion of Recombined DNA

LEGE — Wed, 27 Mar 2024 20:48:26 -0500

ExRec (Exclusion of Recombined DNA) is a Python pipeline that implements the four-gamete test to filter out recombined DNA sites from up to thousands of DNA sequence loci. The pipeline consists of five standalone applications: the first two convert folders of NEXUS or PHYLIP files into the standard input file for the main program that conducts the four-gamete filtering procedures. The pipeline outputs recombination-filtered data in concatenated NEXUS and PHYLIP formats and a tab-delimited table containing descriptive statistics for all loci and the results. This software also allows the user to output the longest non-recombined sequence blocks from loci (current best practice) or randomly select non-recombined blocks from loci (a newer approach). Two other applications in the package convert the recombination-filtered data into single-locus NEXUS or PHYLIP files. The ExRec package can thus facilitate species delimitation, species tree, and historical demography studies by providing loci that better meet the no-recombination assumption in coalescent-based analyses.

Link to the article: https://academic.oup.com/bioinformaticsadvances/article/3/1/vbad174/7455250?searchresult=1

Link to the software:
https://github.com/Sammccarthypotter/ExRec

Address of the bookmark: https://github.com/Sammccarthypotter/ExRec

Keep Your Important SSH Session Running when You Disconnect from Server !!!

Jitendra Narayan — Sat, 15 Mar 2014 21:39:17 -0500

As a Bioinformatician/ Computational biologist we swim in the ocean of genomic/proteomics data, and play with them with an ease. In our day to day simulation, analysis, comparative study we do need to run exhaustive programs, which might take more than a week. In such cases we do need to disconnect from sever in a way that our program/session should not get terminated. To do so there are lots of software, tools such as tmux ( http://tmux.sourceforge.net/, nohup (http://ss64.com/bash/nohup.html) , byobu (https://help.ubuntu.com/10.04/serverguide/byobu.html) and other commands (disown -a && exit), but following are the ones I use the most.

Screen is like a window manager for your console. It will allow you to keep multiple terminal sessions running and easily switch between them. It also protects you from disconnection, because the screen session doesn’t end when you get disconnected.

You’ll need to make sure that screen is installed on the server you are connecting to. If that server is Ubuntu or Debian, just use this command:

sudo apt-get install screen

Now you can start a new screen session by just typing screen at the command line. You’ll be shown some information about screen. Hit enter, and you’ll be at a normal prompt.

To disconnect (but leave the session running)

Hit Ctrl + A and then Ctrl + D in immediate succession. You will see the message [detached]

To reconnect to an already running session

screen -r

To reconnect to an existing session, or create a new one if none exists

screen -D -r

To create a new window inside of a running screen session

Hit Ctrl + A and then C in immediate succession. You will see a new prompt.

To switch from one screen window to another

Hit Ctrl + A and then Ctrl + A in immediate succession.

To list open screen windows

Hit Ctrl + A and then W in immediate succession

DNA in Pieces, Survival in Progress: The Rotifer’s Incredible Secret

Jitendra Narayan — Fri, 18 Sep 2026 03:07:47 -0500

A creature smaller than a grain of sand has discovered a surprising method of surviving damage that can break its DNA apart. It lives in the mosses and lichens, in the thin films of water that show up after rain and disappear when the area dries out. Although it has no hard armor, no sharp claws, and no venom, it is still able to survive even when its chromosomes are shattered by radiation and can pass on some of that damaged DNA to its offspring so that the repair process can go on for several generations. The animal in question is a bdelloid rotifer named Adineta vaga, and its unusual method of survival is providing scientists with new ideas regarding how DNA damage can be repaired.

Bdelloid rotifers are small animals well known for their ability to survive in harsh conditions. They can endure extreme dehydration by entering a state of dormancy and then revive when water becomes available again. Scientists have always been interested in the degree to which these animals resist radiation, since radiation can severely damage DNA by causing both of its strands to break. Interestingly, Adineta vaga's ability to survive radiation may be connected to the way in which it tolerates drying out, as dehydration can also damage DNA. The rotifer probably evolved strong DNA protection and repair mechanisms primarily as a response to its severe environment. Its resistance to radiation is merely a byproduct of adapting to the cycles of drying and rehydration.

In order to find out more about this ability, the researchers subjected A. vaga to proton radiation and then examined what effect it had on its chromosomes. The radiation caused serious damage, resulting in hundreds of breaks in the DNA. For the vast majority of organisms, such extensive chromosome damage would be disastrous; it's similar to going through an instruction manual in a shredder and ending up with a large number of separate fragments. The scientists believed that the rotifers would repair their chromosomes before producing offspring. However, what they discovered was unexpected: some of the broken pieces of chromosome did survive and were passed on to the next generations, where the repair process continued.

The discovery took on even greater interest when the scientists examined the offspring of the irradiated rotifers. Rather than observing the same DNA damage in all the generations, they found different sized sections missing and evidence that some of the genetic regions which had been lost were being slowly restored. The researchers proposed that the rotifers use a repair mechanism known as Break-Induced Homologous Extension Repair, or BIHER. In simple terms, a broken piece of DNA can locate a matching sequence on the other chromosome and use it as a guide to rebuild itself. If the damage is beyond what can be repaired in one generation, the repair process can continue in the next.

What makes the rotifer's method of repairing DNA so unusual is that, normally, we consider DNA repair to take place immediately after damage has occurred, with the cell rapidly repairing the damaged DNA. In A. vaga, however, the repair process can be a lot longer. Certain pieces of chromosome survive the initial damage, are inherited, and then serve as the basis for further repairs in later generations. It's as if one generation begins mending a broken book, the next one continues the work, and another eventually completes the job until all the missing parts have been restored.

One reason why this might work is that bdelloid rotifers have holocentric chromosomes. Rather than having a single point, as most chromosomes do, which controls movement during cell division, the ability to move is distributed along the entire length of their chromosomes. As a result, fragments of chromosomes are sometimes able to retain the parts that are necessary for them to move and divide, and this may account for how broken pieces of chromosome can last long enough to be passed on and then repaired.

This finding could have implications that go beyond those concerning rotifers. Scientists are interested in understanding how genomes cope with severe chromosome damage, for example in the case of chromothripsis, where chromosomes break and are then reassembled in new configurations. Investigating A. vaga could help us to learn how cells manage large genetic issues. Moreover, its high resistance to radiation might also prove useful in research into extreme environments, such as the radiation hazards encountered on long space missions. However, these are ideas for future investigation; at present the rotifer's DNA repair mechanism cannot be used to protect people from radiation.

What is perhaps the most interesting aspect of this discovery is that the rotifer's remarkable ability could have developed for a very simple reason. Since an animal living in moss constantly undergoes periods of drying out and then getting wet again, the same mechanisms that protect its DNA when it is dehydrated might also enable it to survive radiation which would damage most other animals. Therefore, the rotifer probably did not evolve solely in order to resist radiation; instead, it became extremely good at coping with the difficult conditions it encounters, and radiation resistance was just a byproduct.

The case of Adineta vaga demonstrates that survival does not always mean avoiding damage; on some occasions it involves being able to recover from damage which appears to be impossible to repair. This small animal can survive having its chromosomes broken into many pieces, retain some of those pieces, pass them on to its offspring, and continue to repair them over successive generations. Although we generally regard DNA as something fragile that must always remain intact, the rotifer indicates an alternative possibility: perhaps a genome does not have to be perfect in order to survive, since sometimes the pieces last long enough for life to reassemble them, bit by bit, generation after generation.

Ream more at https://www.science.org/doi/10.1126/sciadv.aee0891

Check the Size of a directory & Free disk space.

Jitendra Narayan — Mon, 17 Mar 2014 02:35:32 -0500

The amount of databases we bioinformatician deal are just HUGE … In such cases, we always need to check our server for free spaces etc. I planned this article to explains 2 simple commands that most bioinformatician want to know when they start using Linux / BioLinux. First: Size of a directory (du) and and second: free disk space that exists on your machine (df).

'du' – Check the size of a directory

$ du
This command ( du) gives you a list of directories that exist in the current working directory along with their sizes in kilobytes (default). The last line of the output gives you the total size of the current directory including its subdirectories.

$ du /home/jin1
The above command would give you the directory size of the directory /home/david

$ du -h
The same “du”command with some flag gives you a better output than the default one. The option '-h' stands for human readable format. Therefore, in order to print the sizes of the files / directories in your desire notation use this time suffixed with a 'k' if its kilobytes and 'M' if its Megabytes and 'G' if its Gigabytes.

$ du -ah
If you are interested in checking everything present in a folder use above mentioned command. It gives us not only the directories but also all the files that are present in the current directory. The “-a” flag displays the filenames along with the directory names in the output.

$ du -c
This gives you a grand total as the last line of the output. So if your directory occupies 30MB the last 2 lines of the output would be 30M.

$ du -s
Use this command to displays a summary of the directory size. It is the simplest way to know the total size of the current directory.

$ du -S
This would display the size of the current directory excluding the size of the subdirectories that exist within that directory. So it basically shows you the total size of all the files that exist in the current directory.

$ du --exculde=mp3
Several times it required to exclude some directory in our size calculation. In such cases the above command would display the size of the current directory along with all its subdirectories, but it would exclude all the files having the given pattern present in their filenames.

'df' - finding the disk free space / disk usage

$ df
Hmmm … now “df” command is really useful, and I guess you are going to use it over time. Typing the above command, outputs a table consisting of 6 columns. All the columns are very easy to understand. Remember that the 'Size', 'Used' and 'Avail' columns use kilobytes as the unit. The 'Use%' column shows the usage as a percentage which is also very useful.

$ df -h
Displays the same output as the previous command but the '-h' indicates human readable format. Hence instead of kilobytes as the unit the output would have 'M' for Megabytes and 'G' for Gigabytes.

Example: Linux installed on /dev/hda1
$ df -h | grep /dev/hda1

All right, this is not the only option to check the sizes and free spaces but there are a few more options that can be used with 'du' and 'df' . I will discuss it later.

Public Databases for Bioinformatics !

Jit — Tue, 23 Mar 2021 05:32:15 -0500

https://www.nature.com/articles/s41467-020-17155-y

Server Infrastructure:

File Server:

dhara: Synology 3614 Storage Appliance
4 Core Xeon
108TB disk storage
10Gb ethernet to SCG3
Access atx: dhara:5000
Has btsync server (try it - its much better than dropbox)

Compute Servers:

nandi: Kundaje and Phi Server
24 intel cores
256GB RAM
500GB of SSD storage 
36TB RAID6 local storage
4 Intel Phi's (space for 4 more GPU's)


durga: Montgomery and sensitive data
24 intel cores
256GB RAM
500GB of SSD RAID0 storage 
60TB RAID6 local storage

mitra: Bassik and Web/DB Server
24 core
256GB RAM 
500GB of SSD RAID0 storage 
36TB RAID6 local storage

vayu: Kundaje GPU server
4 core
64GB RAM 
200GB of SSD storage 
8TB RAID10 local storage
4 Nvidia GTX 970 4GB GPUs

amold: Bickel and SGE server
32 AMD core
128GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

wotan: Bickel and SGE server
64 AMD core
256GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

Filesystem:

/users/$USER
default home directory
full backups nightly 
nfs mount to dhara
should store code, papers, and other highly processed data here

/mnt/data/
globally accessible data
should store common data here
e.g. genomes and indexes, annotations, ENCODE data  
if you dont want this to count towards your quote you must chown

/mnt/lab_data/$LAB/
lab accessible data
should store lab project data here 
e.g. ATAC-seq prediction data, enhancer prediction, motif calls

/srv/scratch/$USER
fast local storage
not backed up, but on raid and data will never be deleted
most analysis should be performed here

/srv/persistent/$USER
fast local storage
synced nightly, but not backed up
       ie if the hard drives fail or you delete something and notice 
       within 24 hours we can recover. Otherwise not. (vs home which is 
       properly backed up )  
intermediate analysis products that would be hard to recover should be stored here 
       e.g. stochastic analysis results that need to be kept so that paper 
       results can be reproduced

/srv/www/$LABNAME/
web accessible from mitra.stanford.edu
*NOT BACKED UP*

Some parallel programming patterns:

# gzip a bunch of files
parallel gzip -- *.FILESTOGZIP

# fork example in python:
(for more detailed examples look at 
 https://github.com/nboley/grit/ grit/lib/multiprocessing_utils.py)

import os
import time
import random

import multiprocessing

class ProcessSafeOPStream( object ):
    def __init__( self, writeable_obj ):
        self.writeable_obj = writeable_obj
        self.lock = multiprocessing.Lock()
        self.name = self.writeable_obj.name
        return
    
    def write( self, data ):
        self.lock.acquire()
        self.writeable_obj.write( data )
        self.writeable_obj.flush()
        self.lock.release()
        return
    
    def close( self ):
        self.writeable_obj.close()

def worker(queue, ofp):
    # Try without this
    random.seed()
    while True:
        i = queue.get()
        if i == 'FINISHED': return
        # simulate an expensive function
        x = random.random()
        time.sleep(x/10)
        print i, x
        ofp.write("%i\t%s\n" % (i, x))

NSIMS = 10000
NPROC = 25

# populate queue
todo = multiprocessing.Queue()
for i in xrange(NSIMS): todo.put(i)
for i in xrange(NPROC): todo.put('FINISHED')

ofp = ProcessSafeOPStream( open("output.txt", "w") )

pids = []
for i in xrange(NPROC):
    pid = os.fork()
    if pid == 0:
       worker(todo, ofp)
       os._exit(0)
    else:
       pids.append(pid)  

for pid in pids:
    os.waitpid(pid, 0)

ofp.close()

print "FINISHED"

For use case 1 we obtained the following ENCODE and ROADMAP datasets https://www.encodeproject.org/files/ENCFF446WOD/@@download/ENCFF446WOD.bed.gz, https://www.encodeproject.org/files/ENCFF546PJU/@@download/ENCFF546PJU.bam, https://www.encodeproject.org/files/ENCFF059BEU/@@download/ENCFF059BEU.bam. Blacklisted regions were obtained from http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/hg38-human/hg38.blacklist.bed.gz. The human genome version hg38 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz.

For use case 2 we used the set of narrowPeak files summarized in https://github.com/wkopp/janggu_usecases/tree/master/extra/urls.txt (archived version v1.0.1). The human genome version hg19 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/hg19.fa.gz

For use case 3 we used the ENCODE datasets https://www.encodeproject.org/files/ENCFF591XCX/@@download/ENCFF591XCX.bam, https://www.encodeproject.org/files/ENCFF736LHE/@@download/ENCFF736LHE.bigWig, https://www.encodeproject.org/files/ENCFF177HHM/@@download/ENCFF177HHM.bam as we as the GENCODE annotation v29 from ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_29/gencode.v29.annotation.gtf.gz.

Address of the bookmark: http://mitra.stanford.edu/

JRF at Gautam Buddha University

Thu, 27 Mar 2014 03:53:57 -0500

Gautam Buddha University (GBU) Noida invites applications for the follow posts
2014 March Advertisement from Gautam Buddha University (GBU)
Junior Research Fellow (JRF)
No. of Positions: 01
Educational Qualifications:
Master degree in any discipline of Life Science with NET qualified or valid GATE score. Desirable Qualification: Preference will be given to candidates having research experience in Bioinformatics
Experience:

(details of experience required)
Pay Scale:
INR Rs.12000/-P.M. + HRA
Category:
Science and Research Jobs
How To Apply:
The interested candidates should report for the Interview on 31st
March, 2014 at 10:00 am in the Conference Room of Dean, School of Biotechnology, First floor, Gautam Buddha University, Greater
Noida. Interested candidates may also send their resume to undersigned by post-mail/e-mail shaktis@gbu.ac.in or shaktisahi@gmail.com. No TA and DA will be paid for appearing for the interview
Download Official Notification:

http://www.gbu.ac.in/Recruitment/JRF_advertisement_DSTProject_Shakti_24March14.pdf

Junior Research Fellowship at G.B. PANT UNIVERSITY

Thu, 03 Apr 2014 12:29:46 -0500

DEPARTMENT OF MOLECULAR BIOLOGY & GENETIC ENGINEERING
COLLEGE OF BASIC SCIENCE AND HUMANITIES
G.B. PANT UNIVERSITY OF AGRICULTURE AND TECHNOLOGY
PANTNAGAR -263145, UTTARAKHAND

No. CBSH/MBGE/356

Subject: Advertisement for the award of Junior Research Fellowship.

Applications are invited for award of one Junior Research Fellowship on a consolidated fellowship of Rs. 12,000/- pm in the project “Bioinformatics Sub-DIC ”, under the Coordinatorship Dr. Anil Kumar. The fellowship is purely temporary and may continue till the duration of the project or maximum three years which ever is earlier. The appointment shall be given on six monthly review basis.

ESSENTIAL QUALIFICATION

M.Sc. Bioinformatics having research experience on In silico experimentation.

Candidates possessing the above qualifications may submit their application on
plain paper in the following format to the undersigned latest 18 April, 2014 the interviews will be held on 19 April, 2014 at 11.00 AM in the office of the undersigned. No separate letter for interview will be issued or any TA/DA will be paid for attending the interview.

Advertisement: http://www.gbpuat.ac.in/01042014_18april14_Advertisement%20for%20JRF%20Position,%20BI.pdf

Bandage: interactive visualization of de novo genome assemblies

Shruti Paniwala — Mon, 04 Dec 2017 10:09:37 -0600

Bandage (a Bioinformatics Application for Navigating De novo Assembly Graphs Easily) is a tool for visualizing assembly graphs with connections. Users can zoom in to specific areas of the graph and interact with it by moving nodes, adding labels, changing colors and extracting sequences. BLAST searches can be performed within the Bandage graphical user interface and the hits are displayed as highlights in the graph. By displaying connections between contigs, Bandage presents new possibilities for analyzing de novo assemblies that are not possible through investigation of contigs alone.

Availability and implementation: Source code and binaries are freely available at https://github.com/rrwick/Bandage. Bandage is implemented in C++ and supported on Linux, OS X and Windows. A full feature list and screenshots are available at http://rrwick.github.io/Bandage.

Address of the bookmark: http://rrwick.github.io/Bandage/

Bioinformatics JRF/SRF position at University of Hyderabad

Tue, 15 Apr 2014 20:07:52 -0500

UNIVERSITY OF HYDERABAD SCHOOL OF LIFE SCIENCES

Applications are invited from qualified individuals for a JRF/SRF position (sponsored by DBT/DST) at Prof. Jagan Pongubala’s laboratory, University of Hyderabad. Dr. Pongubala’s laboratory is investigating the molecular pathways that control the development of innate and adaptive immune cell types utilizing a combination of genetic, molecular and computational approaches.

JRF/SRF

Masters degree in Bioinformatics (M.Sc./M.Tech.)

Rs. 12,000+HRA
Rs. 16,000+HRA

Initial appointment is for one year and subjected to renewal up to 2 years

Candidates selected for the above position would have a choice to work on computational biology or experimental biology. Candidates interested to work on computational biology are expected to perform high-throughput sequencing (NGS) data analysis and should have a strong background in Bioinformatics & Computational Biology, good programming skills particularly Perl, Python, R and work experience in Linux environment.

Candidates interested to work on experimental biology should have work experience in techniques that are routinely used in molecular biology and mammalian cell culture. A basic knowledge of bioinformatics is also desired.

Applicants for the above positions should have a Masters degree (M.Tech/M.Sc) with an aggregate marks greater than 70% or a 7.5 CGPA. Candidates having JRF-fellowship through CSIR/UGC/ICMR/DBT will be encouraged to enroll into Ph.D. program. The interested candidates having excellent organizational skills and the ability to work in a team environment with an aspiration to learn new techniques and explore new scientific areas are requested to generate their resume using the link https://cvmkr.com/CV/new#0 and forward to pongubalajagan@gmail.com

Review of applications will begin immediately and continue until the position is filled. Eligible candidates will be called for an interview. No TA/ DA will be paid for attending the interview or at the time of joining the post. Applicants should note that the appointment is purely temporary and subjected to renewal up to three years and there is no Right to Claim for any regular appointment with the University.

Corresponding address: Jagan Pongubala, Ph.D.
Department of Animal Sciences
School of Life Sciences, Room:S44
University of Hyderabad
Gachibowli, Hyderabad 500046

Advertisement: https://www.uohyd.ac.in/images/recruitment/jrf-srf_130414.pdf