http://gmod.org/wiki/Diya

Address of the bookmark: https://sourceforge.net/projects/diyg/

Genomes and genomics:

Bioinformatics and Genomics:

Structural genomics tutorial:

Comparative Genomics Tutorial:

GENOME TUTORIAL:

Tools and resources for identifying protein families, domains and motifs

Bioinformatics Tools 
Tips, Tutorials, and Terminology for Using Selected Resources in Genome Database Guide:

A Web-Based Comparative Genomics Tutorial for Investigating Microbial Genomes:

Free Online Tutorials Teach Anyone How to Use Genome Databases:

Circos to create concise, explanatory, unique and print-ready visualizations of your data:

Genomics and Comparative Genomics Learning Module:

Computational Challenges in Comparative Genomics

A Tutorial:

A Comparative Genomics Resource for Grains:

PLAZA: A Comparative Genomics Resource to Study Gene and Genome Evolution in Plants:

VISTA:

Software for Genomics

1. Artemis Artemis is a free genome viewer and annotation tool that allows visualization of sequence features and the results of analyses within the context of the sequence, and its six-frame translation.
2. Chromas It will display and prints chromatogram files from ABI automated DNA sequencers, and Staden SCF files which the analysis programs for ALF, Li-Cor and Visible Genetics OpenGene sequencers can create.
3. Glimmer A system for finding genes in microbial DNA, especially the genomes of bacteria and archaea.Glimmer (Gene Locator and Interpolated Markov Modeler) uses interpolated Markov models (IMMs) to identify the coding regions and distinguish them from noncoding DN
4. Glimmer HMM A fast and accurate gene finder based on a GHMM architecture, developed specifically for eukaryotes. It incorporates splice site models adapted from the GeneSplicer program and uses interpolated Markov models for evaluating the coding regions.
5. Glimmer M A gene finder derived from Glimmer, but developed specifically for eukaryotes. It is based on a dynamic programming algorithm that considers all combinations of possible exons for inclusion in a gene model and chooses the best of these combinations. The d
6. MUMmer MUMmer is a system for rapidly aligning entire genomes, whether in complete or draft form.
7. pDRAW pDRAW32 is being developed as a free time hobby project. It is far from finished, but as it has reached a point where it could be helpful for many labs, it is now available to the scientific community.
8. Sequin Sequin is a stand-alone software tool developed by the NCBI for submitting and updating entries to the GenBank, EMBL, or DDBJ sequence databases. It is capable of handling simple submissions that contain a single short mRNA sequence, and complex submissio
9. Staden The Staden Package consists of a series of tools for DNA sequence preparation (pregap4), assembly (gap4), editing (gap4) and DNA/protein sequence analysis (spin).

For more software @ http://bioinformaticsonline.com/bookmarks/view/926/list-of-popular-bioinformatics-softwaretools

1. Must have basic understanding of molecular biology and Genomics.
2. Experience in application development or must have expertise in programming using either of Perl/Python.
3. Experience in statistical programming using R/Bioconductor/Matlab.
4. Strong concept in statistical and mathematical modelling.
5. Experience in designing and developing the bioinformatics pipeline.
6. Must have minimum 2+ years of hands on experience in NSG data analysis such as RNA-Seq,Exome-Seq ,Chip-Seq and downstream analysis.
7. Knowledge in WGS ,WES, Targeted re-sequencing,GWAS and population genomics will be preferred.
8. Must have experience working on opensource software/Framework and commercial software for NGS data analysis and reporting.
9. Should be aware of handling big data and guiding team members on multiple projects simultaneously.
10. Should have experience coordinating with different groups of clinical research scientist for various project requirements.
11. Ability to work as team as well as independently with minimal support.

More at http://www3.ocimumbio.com/

Research with the help of bioinformatics with a trans-disciplinary approach is yielding good results.

http://www.thehindu.com/features/education/research/research-with-help-of-bioinformatics-helpful/article2295629.ece

Perl's second wave of adoption came from the growth of the world wide web. Dynamic web pages—the precursor to modern web applications—were easy to create with Perl and CGI. Thanks to Perl's ubiquity as a language for system administrators and its power to manipulate text, it was the default choice for web programming. Its presence everywhere made it popular and, in some ways, the duct tape of the Internet.

Web Application Development

The old days of CGI programs and the simple development style that represented seem clunky. Web pages have become web applications. Development has moved from generating static HTML to both client and server side programming, with rich client interfaces and powerful backends.

Perl is still well suited for developing modern web apps. The language grows more powerful and easier to use every year, the available libraries are wonderful and keep getting better, and the inventions and discoveries available in modern Perl are unsurpassed.

In particular, a modern Perl developer can do amazing things with modern Perl tools. If you still think of Perl web development as a cgi-bin directory full of messy scripts that spew warnings to STDERR, you're a decade out of date. Better yet, you can replace that mess piecemeal, thanks to the new tools and techniques of modern Perl. See, for example, the ever-growing list of technologies Built in Perl.

Modern Perl Web Frameworks

While the old wave of web development may have made the CGI.pm module central, modern Perl web programming follows a stricter separation of business logic, URL and request routing, and output. The days of slinging a string here, an array there, a Perl hash yonder, declaring every variable at the top of the program, and maybe making a subroutine are gone. The Perl world has seen the value of abstraction and ways to mechanize away boilerplate. Perl has dozens of frameworks and toolkits designed to make web development and deployment simpler.

Any of a dozen of these frameworks will help you do great things, but three in particular stand out. You can build web sites and web applications of tremendous value with all three. These are neither the only good possibilities (think of POE or Jifty or Continuity or...) nor the only mechanisms for web programming with Perl (see Mechanize or LWP or Mojo::UserAgent for more). Yet if you want three good options to choose between, start here.

Catalyst

The Catalyst framework is a flexible and powerful system for building small to large web apps. It uses the Moose object system to provide great APIs for extension and further development. It's the most mature of the modern top Perl web frameworks, yet it retains its flexibility and vibrancy. In particular, its plugin and extension ecosystem allows it to evolve to provide new and essential features.

Catalyst has embraced the Plack/PSGI standard for Perl web deployment and recent versions are exploring high-scalability, event-based request handling models.

Dancer

The Dancer framework is deliberately minimal in syntax and scope, but it also has a vibrant plugin ecosystem. Dancer particularly excels for smaller sites and applications, though good programmers can build larger things with it.

The first version of Dancer was easy to use. Dancer 2 continues that ease while improving the internals and robustness of applications.

Mojolicious

The Mojolicious (Mojo) framework has a real-time design based on high performance event handling. Its focus is solving new and interesting problems in simple and effective ways, and the project has produced a lot of new code that does old things in better ways.

In particular, Mojolicious goes to great lengths to support new web standards, such as CSS 3, web sockets, and HTTP 2.

Where Catalyst embraces the CPAN fully, Mojolicious by design provides most of what an average app might need in a single download. It's still fully compatible with the CPAN, but the intention is to provide good working defaults in a package that's easy to start with. Mojo's fans are quick to praise it as fun to develop.

A modern Perl web developer should be familiar with at least one of these frameworks.

Modern Perl Storage Mechanisms

Perl's venerable DBI module has been the focal point of database access since its invention. Its design allows it to provide the same interface to huge relational databases and flat files alike through its DBD extension mechanism. Yet the DBI by itself isn't the be-all, end-all of data storage and access in Perl.

DBIx::Class

DBIx::Class sits on top of DBI to provide an API to your database based on the concept of queries and results. This is often sufficient to remove all but the most complicated of SQL from your code, leaving you to manipulate your business models instead of the small details of how a relational database works. The power and maintainability you receive is well the small cost of the learning curve.

Even better, DBIC can manage (and even generate) your database schema for you.

Recent versions of DBIC have demonstrated that a well-written ORM can perform much better than even clever hand-written code. Because it builds on the Perl DBI, it scales everywhere from SQLite to PostgreSQL, MySQL, Oracle, and more.

Rose::DB

The lesser-known but no less powerful Rose::DB::Object builds on Rose::DB to provide an object-relational mapper for Perl. While its high level features most directly compare to those of DBIx::Class, it's often measurably faster.

NoSQL on the CPAN

Of course the CPAN has modules for almost any NoSQL database or job queue or persistence mechanism you could name, and several you have never heard of. Everything you need is a quick CPAN or cpanm away!

Modern Perl Deployment Strategies

In the early days of the web, deploying a Perl web application meant putting one or more .cgi or .pl files in a special directory and hoping that your system administrator had everything configured correctly. The execution model was often slow and cumbersome, and accessing shared resources such as databases was often tricky.

Modern Perl has better choices. While deployment strategies are the source of many arguments, the return on your investment from learning the modern way is impressive.

Plack/PSGI

The PSGI specification (as exemplified by Plack) describes a strategy for building Perl web apps independent of server and with the possibility to share custom processing behaviors.

In other words, it's a standard for writing Perl apps to take advantage of the huge ecosystem of Perl development available on the CPAN without tying yourself to a server like Apache, Apache 2, nginx, or anything else.

Any good modern Perl web framework (including those listed here) supports PSGI. Several deployment mechanisms exist to meet various business needs which also support PSGI. In particular, you can deploy the same application with a local testing server on your own machine as you can to your production server or servers without changing your application at all.

mod_perl

The older but still viable mod_perl Apache httpd module embeds Perl into the web server. This was the first widespread persistence mechanism for Perl web applications themselves and it's still popular to this day, though PSGI compliance is often the choice for new development. (PSGI handlers to use mod_perl as the backend are available.)

Modern Perl developers should familiarize themselves with PSGI and the wealth of available Plack middleware.

Perl Web Development

Of course no discussion of Perl web development would be complete without mentioning the strength of the CPAN. Almost any project will benefit from the wealth of freely available libraries built to solve real problems. These distributions run the gamut from full-blown web frameworks and content management systems to APIs for web services, development tools, testing systems, and interfaces to document formats and external resources.

For example, if you need to write a web service which accepts JSON data and produces Excel spreadsheets, you can glue together a few CPAN distributions and get the job done early. If you need to consume XML from a remote service and emit a PDF, you're in luck.

Perl's prowess as a general purpose programming language as well as its flexibility and power in managing text and gluing systems together make it a wonderful fit for web development. The community's adoption of modern Perl standards such as PSGI and Plack only enhance your power.

Web application development in Perl is still viable, and modern Perl tools and techniques and libraries make it more powerful and pleasant than ever.

• File System
  ls — list items in current directory
  ls -l — list items in current directory and show in long format to see perimissions, size, and modification date
  ls -a — list all items in current directory, including hidden files
  ls -F — list all items in current directory and show directories with a slash and executables with a star
  ls dir — list all items in directory dir
  cd dir — change directory to dir
  cd .. — go up one directory
  cd / — go to the root directory
  cd ~ — go to to your home directory
  cd - — go to the last directory you were just in
  pwd — show present working directory
  mkdir dir — make directory dir
  rm file — remove file
  rm -r dir — remove directory dir recursively
  cp file1 file2 — copy file1 to file2
  cp -r dir1 dir2 — copy directory dir1 to dir2 recursively
  mv file1 file2 — move (rename) file1 to file2
  ln -s file link — create symbolic link to file
  touch file — create or update file
  cat file — output the contents of file
  less file — view file with page navigation
  head file — output the first 10 lines of file
  tail file — output the last 10 lines of file
  tail -f file — output the contents of file as it grows, starting with the last 10 lines
  vim file — edit file
  alias name 'command' — create an alias for a command
  
• System
  shutdown — shut down machine
  reboot — restart machine
  date — show the current date and time
  whoami — who you are logged in as
  finger user — display information about user
  man command — show the manual for command
  df — show disk usage
  du — show directory space usage
  free — show memory and swap usage
  whereis app — show possible locations of app
  which app — show which app will be run by default
  
• Process Management
  ps — display your currently active processes
  top — display all running processes
  kill pid — kill process id pid
  kill -9 pid — force kill process id pid
  
• Permissions
  ls -l — list items in current directory and show permissions
  chmod ugo file — change permissions of file to ugo - u is the user's permissions, g is the group's permissions, and o is everyone else's permissions. The values of u, g, and o can be any number between 0 and 7.
  7 — full permissions
  6 — read and write only
  5 — read and execute only
  4 — read only
  3 — write and execute only
  2 — write only
  1 — execute only
  0 — no permissions
  chmod 600 file — you can read and write - good for files
  chmod 700 file — you can read, write, and execute - good for scripts
  chmod 644 file — you can read and write, and everyone else can only read - good for web pages
  chmod 755 file — you can read, write, and execute, and everyone else can read and execute - good for programs that you want to share
  
• Networking
  wget file — download a file
  curl file — download a file
  scp user@host:file dir — secure copy a file from remote server to the dir directory on your machine
  scp file user@host:dir — secure copy a file from your machine to the dir directory on a remote server
  scp -r user@host:dir dir — secure copy the directory dir from remote server to the directory dir on your machine
  ssh user@host — connect to host as user
  ssh -p port user@host — connect to host on port as user
  ssh-copy-id user@host — add your key to host for user to enable a keyed or passwordless login
  ping host — ping host and output results
  whois domain — get information for domain
  dig domain — get DNS information for domain
  dig -x host — reverse lookup host
  lsof -i tcp:1337 — list all processes running on port 1337
  
• Searching
  grep pattern files — search for pattern in files
  grep -r pattern dir — search recursively for pattern in dir
  grep -rn pattern dir — search recursively for pattern in dir and show the line number found
  grep -r pattern dir --include='*.ext — search recursively for pattern in dir and only search in files with .ext extension
  command | grep pattern — search for pattern in the output of command
  find file — find all instances of file in real system
  locate file — find all instances of file using indexed database built from the updatedb command. Much faster than find
  sed -i 's/day/night/g' file — find all occurrences of day in a file and replace them with night - s means substitude and g means global - sed also supports regular expressions
  
• Compression
  tar cf file.tar files — create a tar named file.tar containing files
  tar xf file.tar — extract the files from file.tar
  tar czf file.tar.gz files — create a tar with Gzip compression
  tar xzf file.tar.gz — extract a tar using Gzip
  gzip file — compresses file and renames it to file.gz
  gzip -d file.gz — decompresses file.gz back to file
  
• Shortcuts
  ctrl+a — move cursor to beginning of line
  ctrl+f — move cursor to end of line
  alt+f — move cursor forward 1 word
  alt+b — move cursor backward 1 word
  

https://academic.oup.com/bioinformatics/article/30/18/2559/2475628
Reference-free SNP detection: dealing with the data deluge

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4083407/
GATB/DiscoSnp: DiscoSnp is designed for discovering all kinds of SNPs (not only isolated ones), as well as insertions and deletions, from raw set(s) of reads.

https://github.com/GATB/DiscoSnp
De novo assembly | Oxford Nanopore Technologies

https://nanoporetech.com/taxonomy/term/131
De novo long-read assembly of a complex animal genome | bioRxiv

https://www.biorxiv.org/content/early/2017/09/10/187054
Rapid de novo assembly of the European eel genome from nanopore sequencing reads | Scientific Reports

https://www.nature.com/articles/s41598-017-07650-6.epdf?author_access_token=dktG7e98wyRJnaEEMTcPqtRgN0jAjWel9jnR3ZoTv0P7E7t-wVGo30iojNO7dICajNY_7PE5xVPv6OoLe7hn9TeUjcZ5umREOzNoPMWkfYH58RS6uxm3vm4e4BG2AA_WKW84i6egKK271NwMq-NfzA%3D%3D
nanoporetech/ont-assembly-polish: ONT assembly and Illumina polishing pipeline

https://github.com/nanoporetech/ont-assembly-polish
Generade-nl/TULIP: TULIP - The Uncorrected Long read Itegration Pipeline

https://github.com/Generade-nl/TULIP
www.nature.com

https://www.nature.com/articles/s41598-017-03996-z
Example gallery of NanoPlot – Gigabase or gigabyte

https://gigabaseorgigabyte.wordpress.com/2017/06/01/example-gallery-of-nanoplot/
Tool documentation

https://broadinstitute.github.io/picard/command-line-overview.html
Chromosome-scale scaffolding of de novo genome assemblies based on chromatin interactions. - PubMed - NCBI

https://www.ncbi.nlm.nih.gov/pubmed/24185095
MAFFT ver.7 - a multiple sequence alignment program

https://mafft.cbrc.jp/alignment/software/algorithms/algorithms.html
Measuring the distance between multiple sequence alignments | Bioinformatics | Oxford Academic

https://academic.oup.com/bioinformatics/article/28/4/495/212883
The MUMmer 3 examples

http://mummer.sourceforge.net/examples/
MAFFT ver.7 - a multiple sequence alignment program

https://mafft.cbrc.jp/alignment/software/tips.html
Omega | Overlap-graph de novo Assembler for Metagenomics

https://omega.omicsbio.org/
abiswas-odu/Disco: Multi-threaded Distributed Memory Overlap-Layout-Consensus (OLC) Metagenome Assembler

https://github.com/abiswas-odu/Disco
SAGE: String-overlap Assembly of GEnomes | BMC Bioinformatics | Full Text

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-15-302

Fast and sensitive mapping of nanopore sequencing reads with GraphMap | Nature Communications

https://www.nature.com/articles/ncomms11307
lumpy-sv/extractSplitReads_BwaMem at master · arq5x/lumpy-sv

https://github.com/arq5x/lumpy-sv/blob/master/scripts/extractSplitReads_BwaMem
jts/nanocorrect: Experimental pipeline for correcting nanopore reads

https://github.com/jts/nanocorrect

video - how to install flash plugin on ubuntu 14.04 LTS 64-bit version - Ask Ubuntu

https://askubuntu.com/questions/469553/how-to-install-flash-plugin-on-ubuntu-14-04-lts-64-bit-version
lh3/fermi: A WGS de novo assembler based on the FMD-index for large genomes

https://github.com/lh3/fermi
Multi-metagenome

http://madsalbertsen.github.io/multi-metagenome/docs/step9.html
Bandage by rrwick

https://rrwick.github.io/Bandage/
Codon Optimization OnLine (COOL): a web-based multi-objective optimization platform for synthetic gene design | Bioinformatics | Oxford Academic

https://academic.oup.com/bioinformatics/article/30/15/2210/2391162
Genome Architecture and Evolution of a Unichromosomal Asexual Nematode - ScienceDirect

https://www.sciencedirect.com/science/article/pii/S096098221731076X?via%3Dihub#fig4
How to determine chimeras in my de novo assembly? - SEQanswers

http://seqanswers.com/forums/showthread.php?t=26721
samtools(1) manual page

http://www.htslib.org/doc/samtools.html
How To Filter Mapped Reads With Samtools

https://www.biostars.org/p/56246/
The MUMmer 3 manual

http://mummer.sourceforge.net/manual/#nucmer
assembly_olc.pdf

http://www.cs.jhu.edu/~langmea/resources/lecture_notes/assembly_olc.pdf
SAM and BAM filtering oneliners

https://gist.github.com/davfre/8596159
Inroduction to dot-plots

http://www.code10.info/index.php%3Foption%3Dcom_content%26view%3Darticle%26id%3D64:inroduction-to-dot-plots%26catid%3D52:cat_coding_algorithms_dot-plots%26Itemid%3D76
RepeatFinder Home Page

http://www.cbcb.umd.edu/software/RepeatFinder/
RepeatFinderReprint.pdf

http://www.cbcb.umd.edu/software/RepeatFinder/RepeatFinderReprint.pdf
https://bernatgel.github.io/karyoploter_tutorial//Tutorial/CreateIdeogram/CreateIdeogram.html

https://bernatgel.github.io/karyoploter_tutorial//Tutorial/CreateIdeogram/CreateIdeogram.html
Circular Visualization in R

http://zuguang.de/circlize_book/book/introduction.html#a-qiuck-glance
Creating a coverage plot using BEDTools and R

https://davetang.org/muse/2015/08/05/creating-a-coverage-plot-using-bedtools-and-r/
Eval: A software package for analysis of genome annotations | BMC Bioinformatics | Full Text

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-4-50
eval-documentation.pdf

http://mblab.wustl.edu/media/software/eval-documentation.pdf
OmicCircos: A Simple-to-Use R Package for the Circular Visualization of Multidimensional Omics Data

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3921174/
sequence - download.tardigrades.org > v1 > sequence

http://download.tardigrades.org/v1/sequence/
ksahlin/BESST: BESST - scaffolder for genomic assemblies

https://github.com/ksahlin/BESST
reubwn/scripts: Useful scripts for various things

https://github.com/reubwn/scripts
ICEberg

http://db-mml.sjtu.edu.cn/ICEberg/index.php
Satsuma - Evolution and Genomics

http://evomics.org/learning/genomics/satsuma/
A complete bacterial genome assembled de novo using only nanopore sequencing data | Nature Methods

https://www.nature.com/articles/nmeth.3444
vezzi/FRC_align: Computes FRC from SAM/BAM file and not from afg files

https://mail.google.com/mail/u/0/#inbox
Read GTF file into R - Dave Tang's blog

https://davetang.org/muse/2017/08/04/read-gtf-file-r/

https://bernatgel.github.io/karyoploter_tutorial//Tutorial/CustomGenomes/CustomGenomes.html

https://bernatgel.github.io/karyoploter_tutorial//Tutorial/CustomGenomes/CustomGenomes.html
Dot: Interactive dot plot for genome-genome alignments

https://dnanexus.github.io/dot/
Zoho Accounts

https://accounts.zoho.eu/signin?servicename=ZohoProjects&serviceurl=https%3A%2F%2Fprojects.zoho.eu%2Fportal%2Favaga2
lh3/minimap2: A versatile pairwise aligner for genomic and spliced nucleotide sequences

https://github.com/lh3/minimap2
SSPACE-LongRead: scaffolding bacterial draft genomes using long read sequence information | BMC Bioinformatics | Full Text

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-15-211
Palindromic gene amplification — an evolutionarily conserved role for DNA inverted repeats in the genome | Nature Reviews Cancer

https://www.nature.com/articles/nrc2591
bioinformatics - BLAST DNA Sequences Reversed - Biology Stack Exchange

https://biology.stackexchange.com/questions/8160/blast-dna-sequences-reversed
LASTZ

http://www.bx.psu.edu/miller_lab/dist/README.lastz-1.02.00/README.lastz-1.02.00a.html
SOGo - (1652) Inbox

https://sogo.unamur.be/SOGo/so/jnarayan/Mail/view
Tetra-Nucleotide Analysis (TNA) | BIOiPLUG Help center

http://help.bioiplug.com/tetra-nucleotide-analysis-tna/

Clustering metagenomic contigs on tetranucleotide frequency — CGAT documentation

http://cgat.readthedocs.io/en/latest/recipes/metagenome_contigs_kmers.html

Pockets Identification

CASTp

Pocket-Finder

PocketPicker

Candidates must have: a PhD degree in statistics, biostatistics or bioinformatics, extensive experience in analyzing high-dimensional data (microarray, SNP, CNVs) and of validation approaches. In addition, experience in penalized regression methods, data base manipulation; and strong programming skills in order to conduct Monte Carlo studies and applications (R). Candidate must have excellent communication skills (verbal, written and presentation), a strong proficiency in Linux system.

This position is available immediately and will be filled as soon as possible. Appointment could be extended beyond the first year based on additional funding.

For more information about the Department of Biostatistics and Bioinformatics, please visit our website: http://www.biostat.duke.edu.

For more info: http://biostat.duke.edu/sites/biostat.duke.edu/files/Halabi%20-%20Postdoc%20Job%20Posting%202013%20updated.pdf

Duke University is an Equal Opportunity/Affirmative Action Employer.

Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.