Dated : 22 Nov 2013 -23 Nov 2013

http://www.ngsasia-congress.com/

Pockets Identification

CASTp

Pocket-Finder

PocketPicker

Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

The Taverna suite is written in Java and includes the Taverna Engine(used for enacting workflows) that powers both the Taverna Workbench(the desktop client application) and the Taverna Server (which allows remote execution of workflows). Taverna is also available as a Command Line Tool for a quick execution of workflows from a terminal.

Address of the bookmark: http://www.taverna.org.uk/

Currently, BETSY consists of modules required for the microarray and next-generation sequencing data [4] such as expression analysis, classification, peak calling, and visualization.

Address of the bookmark: https://github.com/jefftc/changlab

Gargi believes in its mission statement that every student who passes through the portals of the college emerges as a wholly developed individual symbolizing the spirit of enterprise and inquiry that characterizes Gargi.

Bioinformatics Infrastructure Facility (BIF), Gargi College, University of Delhi invites candidates for filling up the following purely temporary positions sponsored by DBT, New Delhi.

1. Name of the post: Traineeship
Essential Qualification: Post Graduate degree in Bioinformatics or any other branch of Life Sciences preferably with dissertation in Bioinformatics.
Desirable Qualification: Prior knowledge of programming languages such as C, VB, SQL etc. and software/database development.

2. Name of the post: Research Associate
Essential Qualification: PhD in Bioinformatics/Biological Sciences/Computer Science or allied sciences with proven experience in bioinformatics.

3. Name of the post: Studentship
Essential Qualifications: Final year Post Graduate students pursuing a degree in Bioinformatics or any branch of Life Science with knowledge of bioinformatics.

How to apply:
Interested candidates are required to appear for the walk in interview on 29th Aug, 2013 at 10.00 AM in Principal’s Office, Gargi College, Sirifort Road, N. Delhi-110049, with their CVs, original documents and a set of Photostat copies of all original documents.

http://www.du.ac.in/fileadmin/DU/students/Pdf/du/advt/2013/16082013_Gargi_RAplus2_Advt.pdf

Interested applicants must submit CV and addresses of three references to ataulfo@unam.mx.

Tenure Track position in Genomic Sciences

Laboratorio Internacional de Investigación sobre el Genoma Humano, UNAM Juriquilla, Querétaro, México

The International Laboratory for Human Genome Research, LIIGH-UNAM (www.liigh.unam.mx) offers a tenure-track position at the level of Assistant Professor (Investigador Asociado C) to perform research, teaching and formation of human resources in the area of: “Genomics of Mendelian Diseases”

Applicants are expected to have a doctorate degree, postdoctoral experience related to the above mentioned area and a strong track record of peer-reviewed publications. Interested applicants must submit CV, email addresses of three references, and a three-page project to Dr. Rafael Palacios, Coordinator of LIIGH-UNAM (palacios@liigh.unam.mx) before June 21, 2019 ………………………………………………………………

Tenure Track position in Genomic Sciences

Laboratorio Internacional de Investigación sobre el Genoma Humano, UNAM Juriquilla, Querétaro, México

The International Laboratory for Human Genome Research, LIIGH-UNAM (www.liigh.unam.mx) offers a tenure-track position at the level of Assistant Professor (Investigador Asociado C) to perform research, teaching and formation of human resources in the area of: “Statistic Population Genomics and its Impact in Complex Diseases”

Applicants are expected to have a doctorate degree, postdoctoral experience related to the above mentioned area and a strong track record of peer-reviewed publications. Interested applicants must submit CV, email addresses of three references, and a three-page statement of research interests to Dr. Rafael Palacios, Coordinator of LIIGH-UNAM (palacios@liigh.unam.mx) before June 21, 2019

#rasalsi #rasa In our Industrial program you will get Knowledge on RNA Seq, CHIP Seq,

Batch Starts From 18^th November 2019

#hurryup #registernow #enquirenow The primary goal of the industrial training program is to provide students with necessary skills making with employable. RASA LSI trains students with the enhanced skills required for them to excel in jobs in biotechnology, pharmaceuticals, BioIT and related industry sectors. At Rasa you will  learn from industry leaders how to apply these skills in life science & have a command over software developing process  by using various methodologies. For Registration visit us on: https://www.rasalsi.com/index.php/front-page/industrial-training/