<![CDATA[BOL: Related items]]> https://bioinformaticsonline.com/related/37586?offset=150 https://bioinformaticsonline.com/pages/view/37590/parallel-processing-with-perl Sat, 25 Aug 2018 11:32:40 -0500 https://bioinformaticsonline.com/pages/view/37590/parallel-processing-with-perl <![CDATA[Parallel Processing with Perl !]]> Here is a small tutorial on how to make best use of multiple processors for bioinformatics analysis. One best way is using perl threads and forks. Knowing how these threads and forks work is very important before implementing them. Getting to know how these work would be really useful before reading this tutorial.

Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

]]> Rahul Nayak https://bioinformaticsonline.com/news/view/140/python-education-material Wed, 10 Jul 2013 16:38:23 -0500 https://bioinformaticsonline.com/news/view/140/python-education-material <![CDATA[Python Education Material]]> Google provide free class for people with a little bit of programming experience who want to learn Python. The google python class includes written materials, lecture videos, and lots of code exercises to practice Python coding. These materials are used within Google to introduce Python to people who have just a little programming experience.

Find more @ https://developers.google.com/edu/python/

]]> Jitendra Narayan https://bioinformaticsonline.com/bookmarks/view/12963/cosmos-our-workflow-management-system-for-ngs-data Wed, 23 Jul 2014 07:29:14 -0500 https://bioinformaticsonline.com/bookmarks/view/12963/cosmos-our-workflow-management-system-for-ngs-data <![CDATA[COSMOS, our workflow management system for NGS data]]> COSMOS, our Python-based management system for implementing large-scale parallel workflows focusing on, but not restricted to, large-scale short-read "NGS" sequencing data is open-access published via Advance Access in Bioinformatics (Gafni et al. 2014).  It is also available for download for non-commercial academic and research purposes at:

 http://cosmos.hms.harvard.edu/.

Address of the bookmark: https://cosmos.hms.harvard.edu/

]]> Jit https://bioinformaticsonline.com/pages/view/35805/python-learning-resources-for-bioinformatics-and-computational-biologist Fri, 02 Mar 2018 06:54:15 -0600 https://bioinformaticsonline.com/pages/view/35805/python-learning-resources-for-bioinformatics-and-computational-biologist <![CDATA[Python learning resources for bioinformatics and computational biologist !]]> Python is a general-purpose language, which means it can be used to build just about anything, which will be made easy with the right tools/libraries.

Professionally, Python is great for backend web development, data analysis, artificial intelligence, and scientific computing. Many developers have also used Python to build productivity tools, games, and desktop apps, so there are plenty of resources to help you learn how to do those as well.

For pros and cons visit http://www.bestprogramminglanguagefor.me/why-learn-python and http://bioinformaticsonline.com/discussion/view/459/python-vs-perl

More resources at https://github.com/CodementorIO/Python-Learning-Resources

Following are the list of useful python programming resources:

]]> Jit https://bioinformaticsonline.com/blog/view/37627/setting-python-version-as-default-on-linux Tue, 04 Sep 2018 10:15:47 -0500 https://bioinformaticsonline.com/blog/view/37627/setting-python-version-as-default-on-linux <![CDATA[Setting python version as default on Linux]]> If you have a later version than 2.6 you'll need to set 2.6 as the default Python. Later versions would be 2.7 and 3.1; see what you have by typing

python -V

at the terminal. For purposes of this example we'll assume you have 3.1 installed. You'll next need to execute the following commands:

 

sudo apt-get install python2.6 idle-python2.6
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.1 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.6 10
sudo update-alternatives --config python

This last command will allow you to choose which version of python to use by default. If you have done everything above correctly, python2.6 should already be set as the default. If it is not, choose it to be the default. From now on, running python should start version 2.6.

Undoing These Changes

In some cases (e.g., installing or updating certain packages), you'll get an error message if you've run the commands above. To update these packages, you'll have to temporarily undo these changes. Here's how to do that:

sudo update-alternatives --remove-all python
sudo ln -s python3.1 /usr/bin/python

Once you're done updating these packages, execute the commands at the top to set python2.6 as the default again.

]]> Rahul Nayak https://bioinformaticsonline.com/bookmarks/view/40834/nucleus-python-and-c-code-for-reading-and-writing-genomics-data Sun, 02 Feb 2020 08:14:19 -0600 https://bioinformaticsonline.com/bookmarks/view/40834/nucleus-python-and-c-code-for-reading-and-writing-genomics-data <![CDATA[Nucleus: Python and C++ code for reading and writing genomics data.]]> Nucleus is a library of Python and C++ code designed to make it easy to read, write and analyze data in common genomics file formats like SAM and VCF. In addition, Nucleus enables painless integration with the TensorFlow machine learning framework, as anywhere a genomics file is consumed or produced, a TensorFlow tfrecords file may be used instead.

Address of the bookmark: https://github.com/google/nucleus

]]> Jit https://bioinformaticsonline.com/bookmarks/view/42923/flanker Sat, 27 Feb 2021 22:04:53 -0600 https://bioinformaticsonline.com/bookmarks/view/42923/flanker <![CDATA[Flanker]]> Flanker, a Python package which performs alignment-free clustering of gene flanking sequences in a consistent format, allowing investigation of mobile genetic elements (MGEs) without prior knowledge of their structure. Flanker can be flexibly parameterised to finetune outputs by characterising upstream and downstream regions separately and investigating variable lengths of flanking sequence.

image

Address of the bookmark: https://github.com/wtmatlock/flanker

]]> Rahul Nayak https://bioinformaticsonline.com/bookmarks/view/34567/jobtree-based-python-wrapper-to-run-the-genome-simulation-tool-suite-evolver Fri, 08 Dec 2017 16:26:32 -0600 https://bioinformaticsonline.com/bookmarks/view/34567/jobtree-based-python-wrapper-to-run-the-genome-simulation-tool-suite-evolver <![CDATA[jobTree based python wrapper to run the genome simulation tool suite Evolver]]> evolverSimControl (eSC) can be used to simulate multi-chromosome genome evolution on an arbitrary phylogeny (Newick format). In addition to simply running evolver, eSC also automatically creates statistical summaries of the simulation as it runs including text and image files. Also included are convenience scripts to: check on a running simulation and see detailed status and logging information; extract fasta sequence files from the leaf nodes of a completed simulation; extract pairwise multiple alignment files (.maf) from leaf and branch nodes from a completed simulation and with the help of mafJoin, join them together into a single maf covering the entire simulation.

Address of the bookmark: https://github.com/dentearl/evolverSimControl

]]> Jit https://bioinformaticsonline.com/bookmarks/view/40099/contiguator Fri, 04 Oct 2019 01:27:58 -0500 https://bioinformaticsonline.com/bookmarks/view/40099/contiguator <![CDATA[CONTIGuator !]]> CONTIGuator is a Python script for Linux environments whose purpose is to speed-up the bacterial genome assembly process and to obtain a first insight of the genome structure using the well-known artemis comparison tool (ACT).

 

Address of the bookmark: https://sourceforge.net/projects/contiguator/

]]> BioStar https://bioinformaticsonline.com/bookmarks/view/44373/mitohifi-a-python-pipeline-for-mitochondrial-genome-assembly-from-pacbio-high-fidelity-reads Tue, 05 Sep 2023 07:31:35 -0500 https://bioinformaticsonline.com/bookmarks/view/44373/mitohifi-a-python-pipeline-for-mitochondrial-genome-assembly-from-pacbio-high-fidelity-reads <![CDATA[MitoHiFi: a python pipeline for mitochondrial genome assembly from PacBio high fidelity reads]]> MitoHiFi v3.2 is a python pipeline distributed under MIT License !

MitoHiFi was first developed to assemble the mitogenomes for a wide range of species in the Darwin Tree of Life Project (DToL)

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-023-05385-y 

Address of the bookmark: https://github.com/marcelauliano/MitoHiFi

]]> Abhi