Solved with perl http://rosalind.info/problems/1a/

#Find the most frequent k-mers in a string.
#Given: A DNA string Text and an integer k.
#Return: All most frequent k-mers in Text (in any order).

use strict;
use warnings;

my $string="ACGTTGCATGTCGCATGATGCATGAGAGCT";
my $kmer=4;
my %myHash;
my $max=0;

for (my $aa=0; $aa<=(length($string)-4); $aa++) {
    my $myStr=substr  $string, $aa,$kmer;
    #print "$myStr\n";
    my $km=kmerMatch ($string, $myStr, $kmer);
    if ($km > $max) { $max = $km;}
    #print "$km\t$myStr\n";
    $myHash{$myStr}=$km;
    
}

#Print all key which have matching values
foreach my $name (keys %myHash){
    print "$name " if $myHash{$name} == $max;
}

sub kmerMatch { #Check the exact matching kmers with sliding window
my ($string, $myStr, $kmer)=@_;
my $count=0;
for (my $aa=0; $aa<=(length($string)-4); $aa++) {
    my $myWin=substr  $string, $aa,$kmer;
    if ($myWin eq $myStr) {
        #print "$myWin eq $myStr\n";
        $count++;
    }
}
return $count;
}

Script are moved to http://bioinformaticsonline.com/snippets/view/34633/clump-finding-problem-solved-with-perl

Address of the bookmark: http://engr.case.edu/li_jing/papers/00798gpattern.pdf

Job description
We are looking for a motivated, self-driven post-doctoral researcher to work with large-scale sequence data analysis. The position is for 24 months and located at Mathematical Statistics, Department of Mathematical Sciences in Erik Kristiansson’s research group. We are focused on methods development for and analysis of next generation DNA sequencing, in particular comparative metagenomics and gene expression analysis (RNA-seq). We have strong interdisciplinary profile and are actively collaborating with several experimental groups, especially within the environmental sciences, ecology, infectious diseases and cancer genomics. More information is available at http://bioinformatics.math.chalmers.se.

The Post-doctoral position is an appointment that offers an opportunity to qualify for further research positions within academia or industry. The majority of your working time is devoted to your own research, normally as a member of a research group. Included in your work is also to take part in supervision of Ph.D. students and M.Sc thesis students. Teaching of undergraduate students may also be included to a small extent.

The employment is limited to a maximum of 2 years (1+1).

Qualifications
The applicant should have Ph.D. degree preferably in bioinformatics, mathematics, statistics, computer science or equivalent by the start of the appointment. Experience from analysis of large-scale data, in particular from next generation DNA sequencing, is highly valued. The applicant should also be proficient in programming (e.g. Python/Java/C) and comfortable with Unix/Linux systems. Interaction with experimental biologists is central and good collaborative skills are therefore important. Fluency in written and spoken English is a strong requirement. As a post-doctoral researcher you are expected to work independently and to be able to supervise/co-supervise PhD and Master’s students.

Application procedure
The application should be marked with Ref 20130126 and written in English. The application should be sent electronically via Chalmers webpage.

Application deadline: September 8, 2013.

For questions, please contact:
Ass prof. Erik Kristiansson, Matematiska Vetenskaper, erik.kristiansson@chalmers.se, +46 31-772 3521, +46 70-5259751.

Chalmers continuously strive to be an attractive employer. Equality and diversity are substantial foundations in all activities at Chalmers.

Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

Xiaole Shirley Liu (lead instructor)
Joshua Starmer
Martin Hemberg
Ting Wang
Feng Yue

Ming Tang
Yang Liu
Jack Kang
Scarlett Ge
Jiazhen Rong
Phillip Nicol
Maartin De Vries

We thank many colleagues in the community, who helped Dr. Liu in prepare the STAT115/215 BIO/BST282 course over the years. 

Address of the bookmark: https://liulab-dfci.github.io/bioinfo-combio/

Centre de Biotechnology de Sfax (CBS):
Bioinformatics activity started at CBS in 2001 with the setting­up of a research and service unit of  bioinformatics. This unit currently includes one senior researcher, one engineer and four Phd  students. Activities include sequence annotation (service) and three research programs: ab initio  prediction of short eukaryote genes, statistical modelling by Bayesian networks approach of signal  transduction pathways and statistical analysis of human sequence variation data (haplotype  reconstruction and linkage disequilibrium). Activities of the Bioinformatics unit could be found at  the website: http://www.cbs.rnrt.tn/ and the research activity report is available under request to  Bioinformatics@cbs.rnrt.tn. Although the computing facilities are good, there is still a need for  trained human resources to strengthen bioinformatics capacities at CBS, particularly in structural  bioinformatics.

Web site and links: http://www.cbs.rnrt.tn