Workshops are conducted in regular intervals on Drug Designing, Protein Modeling and Simulation, Chemoinformatics, Bioinformatics etc.The workshops are highly beneficial for working professionals, students, researcher for enhancements of the skills in short duration.

unzip blatSrc35.zip

to unpack the archive. This gives a directory blatSrc now move into that directory.

#cd blatSrc

before you begin read the README file that comes with the source code.

One thing about building blat is that you need to set the MACHTYPE variable so that the BLAT sources know what type of machine you are compiling the software on.

on most *nix machines, typing

echo $MACHTYPE

will return the machine architecture type.

On my CentOS 6 based system this gave:

x86_64-redhat-linux-gnu

However, what BLAT requires is the 'short value' (ie the first part of the MACHTYPE). To correct this, in the bash shell type (change this to the correct MACHTYPE for your system)

MACHTYPE=x86_64
export MACHTYPE

now running the command:

echo $MACHTYPE

should give the correct short form of the MACHTYPE:

x86_64

now create the directory lib/$MACHTYPE in the source tree. ie:

mkdir lib/$MACHTYPE

For my machine, lib/x86_64 already existed, so I did not have to do this, but this is not the case for all architectures.

The BLAT code assumes that you are compiling BLAT as a non-privileged (ie non-root) user. As a result, you must create the directory for the executables to go into:

mkdir ~/bin/$MACHTYPE

If you are installing as a normal user, edit your .bashrc to add the following (change the x86_64 to be your MACHTYPE):

export PATH=~/bin/x86_64::$PATH

For me, though, this was not good enough. I wanted the executables in /usr/local/bin where all my other code goes. As a result I did some hackery...

There is a master make template in the inc directory called common.mk and I edited this file with the command:

vi inc/common.mk

I replaced the line

    BINDIR=${HOME}/bin/${MACHTYPE}

with

    BINDIR=/usr/local/bin

saved and quit (as this is in my path, I do not need to do anything else)

All the preparation is now done and you can create the blat executables by going into the toplevel of the blat source tree (for me it was /usr/local/src/blat/blatSrc, but change to wherever you unpacked blat into).

Now simply run the command:

make

to compile the code.

Blat installed cleanly and the executables were all neatly placed in /usr/local/bin/x86_64, just like I wanted.

now simply running the command:

blat

on the command line gives me information on blat and sample usage.

Blat is installed and it's installed properly in my system code tree!!!

This is the simple way to use the module

Example1:

use Benchmark;

$first_time = Benchmark->new;

our code……

$second_time = Benchmark->new;

$final_difference = timediff($first_time,$second_time);

print “the code took, timestr($final_difference),”\n”;

that was a very simple way to know the time diff , we can use it to know the time taken by some part of the code in the program.

More sophisticated way:

use Benchmark;
sub first {

my(arguments) = @_;

}

timethese(100, { first => ‘first_sub(arguments)’});

The first argument to timethese is 100 (evaluate 100 times).

Hope this very small tutorial with Benchmark will help people get started.

Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

A list of programs for genotype and SNP calling :

SOAP2 http://soap.genomics.org.cn/index.html

Single-sample High-quality variant database (for example, dbSNP) Package for NGS data analysis, which includes a single individual genotype caller (SOAPsnp)

realSFS http://128.32.118.212/thorfinn/realSFS/

Single-sample Aligned reads Software for SNP and genotype calling using single individuals and allele frequencies. Site frequency spectrum (SFS) estimation

Samtools http://samtools.sourceforge.net/

Multi-sample Aligned reads Package for manipulation of NGS alignments, which includes a computation of genotype likelihoods (samtools) and SNP and genotype calling (bcftools)

GATK http://www.broadinstitute.org/gsa/wiki/index.php/The_Genome_Analysis_Toolkit Multi-sample Aligned reads Package for aligned NGS data analysis, which includes a SNP and genotype caller (Unifed Genotyper), SNP filtering (Variant Filtration) and SNP quality recalibration (Variant Recalibrator)

Beagle http://faculty.washington.edu/browning/beagle/beagle.html

Multi-sample LD Candidate SNPs, genotype likelihoods Software for imputation, phasing and association that includes a mode for genotype calling

IMPUTE2 http://mathgen.stats.ox.ac.uk/impute/impute_v2.html

Multi-sample LD Candidate SNPs, genotype likelihoods Software for imputation and phasing, including a mode for genotype calling. Requires fine-scale linkage map

QCall ftp://ftp.sanger.ac.uk/pub/rd/QCALL

Multi-sample LD ‘Feasible’ genealogies at a dense set of loci, genotype likelihoods Software for SNP and genotype calling, including a method for generating candidate SNPs without LD information (NLDA) and a method for incorporating LD information (LDA). The ‘feasible’ genealogies can be generated using Margarita (http://www.sanger.ac.uk/resources/software/margarita)

MaCH http://genome.sph.umich.edu/wiki/Thunder

Multi-sample LD Genotype likelihoods Software for SNP and genotype calling, including a method (GPT_Freq) for generating candidate SNPs without LD information and a method (thunder_glf_freq) for incorporating LD information

minimap -t 24 assembly.fasta long_reads.fastq.gz | racon -t 24 long_reads.fastq.gz - assembly.fasta racon_assembly.fasta
nucmer -p nucmer assembly.fasta racon_assembly.fasta
show-snps -C -T -r nucmer.delta

This reports Racon's changes in a table. You can exclude indels with the -I option in show-snps. 

This process (Racon -> MUMmer -> SNP table) solves the problem I originally raised in this issue. So as far as I'm concerned, you can close this issue (or keep it open if you still want to implement some kind of variant table).

1. 2 Eggs and 100 Floor Classic Puzzle
2. Five pirates and gold coin Puzzle
3. Six pirates and Gold Coin puzzle
4. Probability of having boy
5. Random Airplane Seats
6. Inverted playing card puzzle
7. Flipping Coins Puzzle
8. Three hat colors Microsoft Puzzle
9. 25 horses 5 tracks Puzzle
10. Gold Bar Puzzle
11. Crossing the Bridge Puzzle
12. Will you accept the bet?
13. The Puzzle of 100 Hats
14. Man fell in Well Puzzle
15. Minimum Number of Weigths
16. One Bulb with 3 Switches
17. Find the minimum number of aircraft
18. Burning ropes to measure time
19. Connect 3 houses with 3 wells
20. Measure 9 minutes from 2 hourglasses puzzle
21. Ant and Triangle Problem
22. The Man in the Elevator
23. Find the survivor
24. Free the prisoners puzzle
25. GREAT STRATEGY CAN ONLY SAVE LIFE

Specially for Microsoft Interview Puzzles, you may refer,
Top 15 Microsoft Interview Puzzles
Microsoft Interview Puzzles

Other MOST COMMON Interview Puzzles-
Top 25 Tech Interview Logical Puzzles

Each of the puzzles got repeated a number of times in interviews even for top tech companies 

➜ bloomfilter git:(master) ✗ swig -Wall -c++ -perl5 BloomFilter.i
➜ bloomfilter git:(master) ✗ g++ -c BloomFilter_wrap.cxx -I/home/urbe/anaconda3/lib/perl5/5.22.0/x86_64-linux-thread-multi/CORE/ -fPIC -Dbool=char -O3
BloomFilter_wrap.cxx:1892:30: fatal error: ../BloomFilter.hpp: No such file or directory
compilation terminated.
➜ bloomfilter git:(master) ✗ cd swig
➜ swig git:(master) ✗ g++ -c BloomFilter_wrap.cxx -I/home/urbe/anaconda3/lib/perl5/5.22.0/x86_64-linux-thread-multi/CORE/ -fPIC -Dbool=char -O3
In file included from BloomFilter_wrap.cxx:1877:0:
../BloomFilter.hpp: In member function ‘void BloomFilter::loadHeader(FILE*)’:
../BloomFilter.hpp:141:59: warning: ignoring return value of ‘size_t fread(void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result [-Wunused-result]
fread(&header, sizeof(struct FileHeader), 1, file);
^
➜ swig git:(master) ✗ g++ -Wall -shared BloomFilter_wrap.o -o BloomFilter.so -O3
➜ swig git:(master) ✗ cd ..
➜ bloomfilter git:(master) ✗ cd ..
➜ lib git:(master) ✗ cd ..
➜ bin git:(master) ✗ ./LINKS
Usage: ./LINKS [v1.8.6]
-f sequences to scaffold (Multi-FASTA format, required)
-s file-of-filenames, full path to long sequence reads or MPET pairs [see below] (Multi-FASTA/fastq format, required)
-m MPET reads (default -m 1 = yes, default = no, optional)
! DO NOT SET IF NOT USING MPET. WHEN SET, LINKS WILL EXPECT A SPECIAL FORMAT UNDER -s
! Paired MPET reads in their original outward orientation <- -> must be separated by ":"
>template_name
ACGACACTATGCATAAGCAGACGAGCAGCGACGCAGCACG:ATATATAGCGCACGACGCAGCACAGCAGCAGACGAC
-d distance between k-mer pairs (ie. target distances to re-scaffold on. default -d 4000, optional)
Multiple distances are separated by comma. eg. -d 500,1000,2000,3000
-k k-mer value (default -k 15, optional)
-t step of sliding window when extracting k-mer pairs from long reads (default -t 2, optional)
Multiple steps are separated by comma. eg. -t 10,5
-o offset position for extracting k-mer pairs (default -o 0, optional)
-e error (%) allowed on -d distance e.g. -e 0.1 == distance +/- 10% (default -e 0.1, optional)
-l minimum number of links (k-mer pairs) to compute scaffold (default -l 5, optional)
-a maximum link ratio between two best contig pairs (default -a 0.3, optional)
*higher values lead to least accurate scaffolding*
-z minimum contig length to consider for scaffolding (default -z 500, optional)
-b base name for your output files (optional)
-r Bloom filter input file for sequences supplied in -s (optional, if none provided will output to .bloom)
NOTE: BLOOM FILTER MUST BE DERIVED FROM THE SAME FILE SUPPLIED IN -f WITH SAME -k VALUE
IF YOU DO NOT SUPPLY A BLOOM FILTER, ONE WILL BE CREATED (.bloom)
-p Bloom filter false positive rate (default -p 0.001, optional; increase to prevent memory allocation errors)
-x Turn off Bloom filter functionality (-x 1 = yes, default = no, optional)
-v Runs in verbose mode (-v 1 = yes, default = no, optional)

Error: Missing mandatory options -f and -s.

ERROR fixed

perl: symbol lookup error: /home/urbe/Tools/LINKS_new/bin/./lib/bloomfilter/swig/BloomFilter.so: undefined symbol: Perl_Gthr_key_ptr

Take a look at the PRICE software from the DeRisi lab. Its meant to do something very similar. http://derisilab.ucsf.edu/index.php?page=software

You could also look at SSPACE (http://www.baseclear.com/landingpages/basetools-a-wide-range-of-bioinformatics-solutions/sspacev12/), ATLAS tools (http://www.hgsc.bcm.tmc.edu/content/bcm-hgsc-software), and SCARPA (http://compbio.cs.toronto.edu/hapsembler/scarpa.html).

See the PAGIT protocol: http://www.sanger.ac.uk/resources/software/pagit/

In particular, take a look at the IMAGE tool: http://genomebiology.com/2010/11/4/R41

Also SOAPdenovo has ha function for scaffolding. Not sure about ABYSS

Here there is a useful explanation of several tools.

https://bioinformaticsonline.com/search?q=scaffolding&entity_type=object&entity_subtype=bookmarks&offset=0&search_type=entities

I could be wrong, but the above answers to your hypothetical scenario appear to miss the point that you aren't interested in assembling the full genome, just the 100 kb part you're interested in. I suggest the following algorithm:

1. Start with the initial assembly C0 of the contigs you have identified as overlapping your region of interest, and the set S of reads those contigs contain. Let C = C0.

2. Repeat:
a. Identify paired-end reads (not in C) for which one or both ends align within, or extending, contigs in C.
b. Identify unpaired reads that align extending these new paired-end reads.
c. Construct a new assembly C' from C and the new reads identified in (a) and (b).
d. Trim C' so it does not extend more than 100 kb to either end of C0. Set C = C'.
e. Let S' denote the reads that contribute to C'. If S' does not contain any reads not present in S, stop. Otherwise, Set S = S'.

3. If you don't have a complete assembly of the region of interest, generate an STS for each end of each contig, probe a library for clones including these STSes, subclone these clones into a paired-end sequencing vector, and generate paired-end reads for this library; then try steps (1) and (2) again, adding these new sequencing reads to what you had before.

4. If your average sequencing depth for the region of interest exceeds 25 or so without filling all gaps, it is likely that the remaining gaps represent sequences that are not getting cloned in your sequencing vectors. Try different sequencing vectors.