Script are moved to http://bioinformaticsonline.com/snippets/view/34633/clump-finding-problem-solved-with-perl

Simple and efficient tools for data mining and data analysis
Accessible to everybody, and reusable in various contexts
Built on NumPy, SciPy, and matplotlib
Open source, commercially usable - BSD license

More at http://scikit-learn.org/stable/index.html

Address of the bookmark: http://scikit-learn.org/stable/auto_examples/index.html

ICRISAT is headquartered in Patancheru near Hyderabad, India, with two regional hubs and five country offices in sub-Saharan Africa. ICRISAT, established in 1972, is a member of the CGIAR Consortium. For more details, see www.icrisat.org.

Responsibilities:Design efficient SQL queries for pulling large sequencing projects.
Serve as a technical adviser to the project leadership and provide computational perspective on product design and deliverability.
Develop and oversee a rapid and incremental software development and release schedule.
Design the software architecture, oversee the implementation and evolution of the design on appropriate hardware platforms.
Working collaboratively in a team environment to design, code, test, debug, and document programs for an integrated genomic analysis pipeline in a rapid and incremental software development and release schedule.
Supervise and review code development and ensure that software products meet project objectives in terms of functionality, scalability, robustness and user experience.
Implement and oversee the QA/QC practices to ensure the development team is adhering to quality standards.
Work closely with the application specialist to integrate feedbacks from teams in each CGIAR center into software customization and improvement.
Assist in training of breeders in the CGIAR centers to use software developed.
Personal Profile:

The applicant should have:

Understanding of genomics data and advanced knowledge of Java, and C/C++ as the programming languages and any of the scripting language like perl and/or Python, SQL
High Performance Computing, data architecture, database platforms and QA/QC practices in software engineering.
She/he should have solid experience in software development projects, preferably as a senior programmer or in the software project management role, and in projects involving big data.
Excellent communication skills are needed to work in this multi-disciplinary, multi-location and multi-cultural team.
Ability to mentor colleagues in quality software development practices is desired.
Educational Qualification : Ph. D or Masters Degree in Computational Biology / Computational Genomics or Equivalent with Research Experience in Mentioned Areas.

More at http://www.icrisat.org/careers/

Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

Essential Qualifications: Ph.D. (Bioinformatics/ Biophysics/ Biotechnology or any other stream of biological/ physical sciences) with a minimum of two publications in reputed peer reviewed journals in the area of structural bioinformatics or biophysics or biomolecular modeling/ simulation.

Job description: Development of bioinformatics tools and algorithms/software for structure based analysis of biomolecular systems. Programmatic access to major biomolecular databases using APIs Knowledge based prediction and analysis of biomolecular structure, function and interactions. Docking/simulations for inhibitor design.

Desirable Qualifications (Research Associate/s): i) Strong computer programming skills (in Python/PERL/PHP or C++ or object oriented database management systems like MySQL etc or scripting languages under LINUX/UNIX environment).

ii) Extensive experience in computational analysis of biomolecular structure/interactions and usage of advanced biomolecular simulation softwares. iii) Adequate knowledge of major databases, webservers and softwares in the area of biomolecular structure/function and drug design. iv) Familiarity with Parallel Programming environments and experience in usage of high-end HPC clusters.

The candidates must highlight their experience in above mentioned fields/topics in their CV. Initial appointment will be for a period of 1 year, subject to extension after review of performance.

Emoluments: As per DST, GOI norms and commensurate with experience.

More at https://www.iisc.ac.in/positions-open/

Biostrings

The Biostrings package from Bioconductor provides an advanced environment for efficient sequence management and analysis in R. It contains many speed and memory effective string containers, string matching algorithms, and other utilities, for fast manipulation of large sets of biological sequences. The objects and functions provided by Biostrings form the basis for many other sequence analysis packages. Documentation

Many of the Perl scripts require that the VCF files are compressed by bgzip and indexed by tabix (both tools are part of the tabix package, available for download here). The VCF files can be compressed and indexed using the following commands

bgzip my_file.vcf
tabix -p vcf my_file.vcf.gz

http://vcftools.sourceforge.net/perl_module.html

Address of the bookmark: http://vcftools.sourceforge.net/perl_module.html

For example ... contig-stats.pl is a Perl script that will automatically describe features of a sequence assembly.

http://milkweedgenome.org/?q=scripts

Address of the bookmark: http://milkweedgenome.org/?q=scripts

#Find the reverse complement of a DNA string.
#Given: A DNA string Pattern.
#Return: Pattern, the reverse complement of Pattern.

use strict;
use warnings;

my $string="AAAACCCGGT";
my $finalString="";
my %hash = (
    "C" => "G",
    "A" => "T",
    "T" => "A",
    "G" => "C",
);

for (my $aa=0; $aa<=(length($string)-1); $aa++) {
    my $char=substr $string, $aa, 1;
    #print $hash{$char};
    $finalString="$hash{$char}"."$finalString";
}

print $finalString;
print "\n";