The most common newbie questions are:

Should I try to use these free open source programs?  Why are we not trying GUI software for computational analysis? Should I use commercial bioinformatics programs/software?”

1. Let’s be open

We generally think free and cheap are useless. But this concept is not applicable when we discuss open source software. Mostly, the bioinformatics software is developed by highly competitive biological programmers who believe in open sharing of knowledge. They come under Open Bioinformatics Foundation or O|B|F which is a non-profit, volunteer run organization focused on supporting open source programming in bioinformatics. The best part about open source tools/software is that they’re free to download the source code and read exactly what the program does. If you are so inclined, you can view all of the parts of the program and see the logical flow of the pipeline. In addition, open source makes an excellent learning tool for any beginning bioinformatician. Moreover, you can modify existing open source programs to deal with cutting-edge problems or to customize your pipeline. Apart from your computational and analysis work, most of the reviewer also prefers the open source based results so that they can validate the results if validation required.

2. Code headache

As a bioinformatician you are supposed to know the basics of programming languages, and if you are not good at it, then please learn it as soon as possible because you are not a bio-analyst but biological programmers. The open source programs usually lack dedicated service and support teams (often because they were the product of an overworked doc/postdoc!) so you are responsible for troubleshooting your own errors most of the time. We commonly receive the HELP email to support and assist to setup the pipeline; you can also find this kind of request on any QA forum. I personally believe this coding horror brings the biggest downside of open-source programs; where you need some programming skills in order to implement the program in your pipeline. But, if you are not able to fix the pipeline and modify the open source code according to your requirements them you should re-think on your bioinformatician name tag!!!

3. Dive into the codes

Some of the biologist turn bioinformatician says “if you can do the same thing with commercial software then why to get migraine with weird codes”, well this statement looks to me that guys are keen to learn swimming but still don’t like to get wet. If you are still using paid software and doing your work by customer support and clicking some of the well-designed GUI button then perhaps you are not interested in learning and trying new and challenging bioinformatics works. You are missing the basic flavour of bioinformatics. Let’s dive into the coding world, I am sure your will enjoy it. I recommend your to swim freely in code’s sea, and enjoy the journey; do not merely watch it from the outside.  

4. Paid does not mean better

The bioinformatics company which are specializes in bioinformatics solutions develop well designed/packed, user friendly software by using a large number of specialised scientist, programmers and support staff. They also provide good services to accomplice your biological analysis work. This means that if you hit a ‘snag’ with your data, help is likely only a phone call away! These companies price their products competitively against the cost of a dedicated bioinformatician. You may be able to afford the program, but not the additional staff! Additionally, most of the functionality that you need in your analysis is already coded into the program. Need to plot a graph? Just click this button right here. It is that easy. But, as a bioinformatician this is not generally well encouraged approach in biological analysis work, because the software is not available to everyone and your data can’t be validated. Moreover, there is very less chances that anyone will repeat your work or love to do similar kind of research (because not all the labs in the world are rich like yours).

5. Take a caution

In biological analysis work, in which you deal GB/TB of data are having maximum chances of getting errors, so please be careful and always cross check your data before coming to any conclusion. Even an error in two line code can alter your entire analysis and display weird results. Some of the scientist blindly believes on commercial software, which is entirely wrong. Using proprietary tools does not absolve you of the need to actually read and research the type of analysis that you are doing. This is particularly true in the case of genome assembly and annotation.

At the end, I would like to tell only one think that open source solutions allows you to do more cutting edge analysis than the commercial tools. So let’s go for it.

Disclaimer:

This is my personal view. I have nothing to do with any company or open source community. The views expressed on these pages are mine alone and not those of my current/past employers. I do reserve the right to remove comments left by spammers or off-topic comments.

• File System
  ls — list items in current directory
  ls -l — list items in current directory and show in long format to see perimissions, size, and modification date
  ls -a — list all items in current directory, including hidden files
  ls -F — list all items in current directory and show directories with a slash and executables with a star
  ls dir — list all items in directory dir
  cd dir — change directory to dir
  cd .. — go up one directory
  cd / — go to the root directory
  cd ~ — go to to your home directory
  cd - — go to the last directory you were just in
  pwd — show present working directory
  mkdir dir — make directory dir
  rm file — remove file
  rm -r dir — remove directory dir recursively
  cp file1 file2 — copy file1 to file2
  cp -r dir1 dir2 — copy directory dir1 to dir2 recursively
  mv file1 file2 — move (rename) file1 to file2
  ln -s file link — create symbolic link to file
  touch file — create or update file
  cat file — output the contents of file
  less file — view file with page navigation
  head file — output the first 10 lines of file
  tail file — output the last 10 lines of file
  tail -f file — output the contents of file as it grows, starting with the last 10 lines
  vim file — edit file
  alias name 'command' — create an alias for a command
  
• System
  shutdown — shut down machine
  reboot — restart machine
  date — show the current date and time
  whoami — who you are logged in as
  finger user — display information about user
  man command — show the manual for command
  df — show disk usage
  du — show directory space usage
  free — show memory and swap usage
  whereis app — show possible locations of app
  which app — show which app will be run by default
  
• Process Management
  ps — display your currently active processes
  top — display all running processes
  kill pid — kill process id pid
  kill -9 pid — force kill process id pid
  
• Permissions
  ls -l — list items in current directory and show permissions
  chmod ugo file — change permissions of file to ugo - u is the user's permissions, g is the group's permissions, and o is everyone else's permissions. The values of u, g, and o can be any number between 0 and 7.
  7 — full permissions
  6 — read and write only
  5 — read and execute only
  4 — read only
  3 — write and execute only
  2 — write only
  1 — execute only
  0 — no permissions
  chmod 600 file — you can read and write - good for files
  chmod 700 file — you can read, write, and execute - good for scripts
  chmod 644 file — you can read and write, and everyone else can only read - good for web pages
  chmod 755 file — you can read, write, and execute, and everyone else can read and execute - good for programs that you want to share
  
• Networking
  wget file — download a file
  curl file — download a file
  scp user@host:file dir — secure copy a file from remote server to the dir directory on your machine
  scp file user@host:dir — secure copy a file from your machine to the dir directory on a remote server
  scp -r user@host:dir dir — secure copy the directory dir from remote server to the directory dir on your machine
  ssh user@host — connect to host as user
  ssh -p port user@host — connect to host on port as user
  ssh-copy-id user@host — add your key to host for user to enable a keyed or passwordless login
  ping host — ping host and output results
  whois domain — get information for domain
  dig domain — get DNS information for domain
  dig -x host — reverse lookup host
  lsof -i tcp:1337 — list all processes running on port 1337
  
• Searching
  grep pattern files — search for pattern in files
  grep -r pattern dir — search recursively for pattern in dir
  grep -rn pattern dir — search recursively for pattern in dir and show the line number found
  grep -r pattern dir --include='*.ext — search recursively for pattern in dir and only search in files with .ext extension
  command | grep pattern — search for pattern in the output of command
  find file — find all instances of file in real system
  locate file — find all instances of file using indexed database built from the updatedb command. Much faster than find
  sed -i 's/day/night/g' file — find all occurrences of day in a file and replace them with night - s means substitude and g means global - sed also supports regular expressions
  
• Compression
  tar cf file.tar files — create a tar named file.tar containing files
  tar xf file.tar — extract the files from file.tar
  tar czf file.tar.gz files — create a tar with Gzip compression
  tar xzf file.tar.gz — extract a tar using Gzip
  gzip file — compresses file and renames it to file.gz
  gzip -d file.gz — decompresses file.gz back to file
  
• Shortcuts
  ctrl+a — move cursor to beginning of line
  ctrl+f — move cursor to end of line
  alt+f — move cursor forward 1 word
  alt+b — move cursor backward 1 word
  

Genomics England will play a key role in building on the UK’s long track record as leader in medical science advances to push the boundaries by unlocking the power of DNA data. The UK will become the first ever country to introduce this technology in its mainstream health system – leading the global race for better tests, better drugs and above all better, more personalised care.

http://www.genomicsengland.co.uk/100k-genome-project/

https://academic.oup.com/bioinformatics/article/30/18/2559/2475628
Reference-free SNP detection: dealing with the data deluge

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4083407/
GATB/DiscoSnp: DiscoSnp is designed for discovering all kinds of SNPs (not only isolated ones), as well as insertions and deletions, from raw set(s) of reads.

https://github.com/GATB/DiscoSnp
De novo assembly | Oxford Nanopore Technologies

https://nanoporetech.com/taxonomy/term/131
De novo long-read assembly of a complex animal genome | bioRxiv

https://www.biorxiv.org/content/early/2017/09/10/187054
Rapid de novo assembly of the European eel genome from nanopore sequencing reads | Scientific Reports

https://www.nature.com/articles/s41598-017-07650-6.epdf?author_access_token=dktG7e98wyRJnaEEMTcPqtRgN0jAjWel9jnR3ZoTv0P7E7t-wVGo30iojNO7dICajNY_7PE5xVPv6OoLe7hn9TeUjcZ5umREOzNoPMWkfYH58RS6uxm3vm4e4BG2AA_WKW84i6egKK271NwMq-NfzA%3D%3D
nanoporetech/ont-assembly-polish: ONT assembly and Illumina polishing pipeline

https://github.com/nanoporetech/ont-assembly-polish
Generade-nl/TULIP: TULIP - The Uncorrected Long read Itegration Pipeline

https://github.com/Generade-nl/TULIP
www.nature.com

https://www.nature.com/articles/s41598-017-03996-z
Example gallery of NanoPlot – Gigabase or gigabyte

https://gigabaseorgigabyte.wordpress.com/2017/06/01/example-gallery-of-nanoplot/
Tool documentation

https://broadinstitute.github.io/picard/command-line-overview.html
Chromosome-scale scaffolding of de novo genome assemblies based on chromatin interactions. - PubMed - NCBI

https://www.ncbi.nlm.nih.gov/pubmed/24185095
MAFFT ver.7 - a multiple sequence alignment program

https://mafft.cbrc.jp/alignment/software/algorithms/algorithms.html
Measuring the distance between multiple sequence alignments | Bioinformatics | Oxford Academic

https://academic.oup.com/bioinformatics/article/28/4/495/212883
The MUMmer 3 examples

http://mummer.sourceforge.net/examples/
MAFFT ver.7 - a multiple sequence alignment program

https://mafft.cbrc.jp/alignment/software/tips.html
Omega | Overlap-graph de novo Assembler for Metagenomics

https://omega.omicsbio.org/
abiswas-odu/Disco: Multi-threaded Distributed Memory Overlap-Layout-Consensus (OLC) Metagenome Assembler

https://github.com/abiswas-odu/Disco
SAGE: String-overlap Assembly of GEnomes | BMC Bioinformatics | Full Text

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-15-302

Fast and sensitive mapping of nanopore sequencing reads with GraphMap | Nature Communications

https://www.nature.com/articles/ncomms11307
lumpy-sv/extractSplitReads_BwaMem at master · arq5x/lumpy-sv

https://github.com/arq5x/lumpy-sv/blob/master/scripts/extractSplitReads_BwaMem
jts/nanocorrect: Experimental pipeline for correcting nanopore reads

https://github.com/jts/nanocorrect

video - how to install flash plugin on ubuntu 14.04 LTS 64-bit version - Ask Ubuntu

https://askubuntu.com/questions/469553/how-to-install-flash-plugin-on-ubuntu-14-04-lts-64-bit-version
lh3/fermi: A WGS de novo assembler based on the FMD-index for large genomes

https://github.com/lh3/fermi
Multi-metagenome

http://madsalbertsen.github.io/multi-metagenome/docs/step9.html
Bandage by rrwick

https://rrwick.github.io/Bandage/
Codon Optimization OnLine (COOL): a web-based multi-objective optimization platform for synthetic gene design | Bioinformatics | Oxford Academic

https://academic.oup.com/bioinformatics/article/30/15/2210/2391162
Genome Architecture and Evolution of a Unichromosomal Asexual Nematode - ScienceDirect

https://www.sciencedirect.com/science/article/pii/S096098221731076X?via%3Dihub#fig4
How to determine chimeras in my de novo assembly? - SEQanswers

http://seqanswers.com/forums/showthread.php?t=26721
samtools(1) manual page

http://www.htslib.org/doc/samtools.html
How To Filter Mapped Reads With Samtools

https://www.biostars.org/p/56246/
The MUMmer 3 manual

http://mummer.sourceforge.net/manual/#nucmer
assembly_olc.pdf

http://www.cs.jhu.edu/~langmea/resources/lecture_notes/assembly_olc.pdf
SAM and BAM filtering oneliners

https://gist.github.com/davfre/8596159
Inroduction to dot-plots

http://www.code10.info/index.php%3Foption%3Dcom_content%26view%3Darticle%26id%3D64:inroduction-to-dot-plots%26catid%3D52:cat_coding_algorithms_dot-plots%26Itemid%3D76
RepeatFinder Home Page

http://www.cbcb.umd.edu/software/RepeatFinder/
RepeatFinderReprint.pdf

http://www.cbcb.umd.edu/software/RepeatFinder/RepeatFinderReprint.pdf
https://bernatgel.github.io/karyoploter_tutorial//Tutorial/CreateIdeogram/CreateIdeogram.html

https://bernatgel.github.io/karyoploter_tutorial//Tutorial/CreateIdeogram/CreateIdeogram.html
Circular Visualization in R

http://zuguang.de/circlize_book/book/introduction.html#a-qiuck-glance
Creating a coverage plot using BEDTools and R

https://davetang.org/muse/2015/08/05/creating-a-coverage-plot-using-bedtools-and-r/
Eval: A software package for analysis of genome annotations | BMC Bioinformatics | Full Text

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-4-50
eval-documentation.pdf

http://mblab.wustl.edu/media/software/eval-documentation.pdf
OmicCircos: A Simple-to-Use R Package for the Circular Visualization of Multidimensional Omics Data

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3921174/
sequence - download.tardigrades.org > v1 > sequence

http://download.tardigrades.org/v1/sequence/
ksahlin/BESST: BESST - scaffolder for genomic assemblies

https://github.com/ksahlin/BESST
reubwn/scripts: Useful scripts for various things

https://github.com/reubwn/scripts
ICEberg

http://db-mml.sjtu.edu.cn/ICEberg/index.php
Satsuma - Evolution and Genomics

http://evomics.org/learning/genomics/satsuma/
A complete bacterial genome assembled de novo using only nanopore sequencing data | Nature Methods

https://www.nature.com/articles/nmeth.3444
vezzi/FRC_align: Computes FRC from SAM/BAM file and not from afg files

https://mail.google.com/mail/u/0/#inbox
Read GTF file into R - Dave Tang's blog

https://davetang.org/muse/2017/08/04/read-gtf-file-r/

https://bernatgel.github.io/karyoploter_tutorial//Tutorial/CustomGenomes/CustomGenomes.html

https://bernatgel.github.io/karyoploter_tutorial//Tutorial/CustomGenomes/CustomGenomes.html
Dot: Interactive dot plot for genome-genome alignments

https://dnanexus.github.io/dot/
Zoho Accounts

https://accounts.zoho.eu/signin?servicename=ZohoProjects&serviceurl=https%3A%2F%2Fprojects.zoho.eu%2Fportal%2Favaga2
lh3/minimap2: A versatile pairwise aligner for genomic and spliced nucleotide sequences

https://github.com/lh3/minimap2
SSPACE-LongRead: scaffolding bacterial draft genomes using long read sequence information | BMC Bioinformatics | Full Text

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-15-211
Palindromic gene amplification — an evolutionarily conserved role for DNA inverted repeats in the genome | Nature Reviews Cancer

https://www.nature.com/articles/nrc2591
bioinformatics - BLAST DNA Sequences Reversed - Biology Stack Exchange

https://biology.stackexchange.com/questions/8160/blast-dna-sequences-reversed
LASTZ

http://www.bx.psu.edu/miller_lab/dist/README.lastz-1.02.00/README.lastz-1.02.00a.html
SOGo - (1652) Inbox

https://sogo.unamur.be/SOGo/so/jnarayan/Mail/view
Tetra-Nucleotide Analysis (TNA) | BIOiPLUG Help center

http://help.bioiplug.com/tetra-nucleotide-analysis-tna/

Clustering metagenomic contigs on tetranucleotide frequency — CGAT documentation

http://cgat.readthedocs.io/en/latest/recipes/metagenome_contigs_kmers.html

Pockets Identification

CASTp

Pocket-Finder

PocketPicker

Candidates must have: a PhD degree in statistics, biostatistics or bioinformatics, extensive experience in analyzing high-dimensional data (microarray, SNP, CNVs) and of validation approaches. In addition, experience in penalized regression methods, data base manipulation; and strong programming skills in order to conduct Monte Carlo studies and applications (R). Candidate must have excellent communication skills (verbal, written and presentation), a strong proficiency in Linux system.

This position is available immediately and will be filled as soon as possible. Appointment could be extended beyond the first year based on additional funding.

For more information about the Department of Biostatistics and Bioinformatics, please visit our website: http://www.biostat.duke.edu.

For more info: http://biostat.duke.edu/sites/biostat.duke.edu/files/Halabi%20-%20Postdoc%20Job%20Posting%202013%20updated.pdf

Duke University is an Equal Opportunity/Affirmative Action Employer.

Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

Candidate:
The candidate is expected to have B.S. or M.S. degree in the disciplines such as Computer Science, Statistics, Applied Mathematics, Physics or Electrical Engineering. The candidates with the backgrounds in Life Science disciplines such as Bioinformatics, Computational or Quantitative Biology will also be considered.

Location:
The position is available at LAPTI (http://lapti.ucc.ie) that is located in the Western Gate Building (http://www.stwarchitects.com/project-information.php?c=1&p=09993) at University College Cork. Western Gate Building Research Complex hosts several UCC departments and provides ideal environment for interdisciplinary research. Cork (sometimes referenced as “Venice of Ireland”) is the second most populous city in the Republic. It has friendly cosmopolitan atmosphere and vibrant culture. A number of American industrial giants such as Apple , EMC and Pfizer have chosen Cork as a home for their European headquarters.

Application process:
The details of the application process are given at http://lapti.ucc.ie/jobs.html. To ensure prompt processing of your application use the subject line: ‘Ph.D. computational’. All applications received prior to August the 1st are guaranteed equal consideration. However, applications at the later dates will also be considered until the position is filled.

Interested applicants must submit CV and addresses of three references to ataulfo@unam.mx.

Tenure Track position in Genomic Sciences

Laboratorio Internacional de Investigación sobre el Genoma Humano, UNAM Juriquilla, Querétaro, México

The International Laboratory for Human Genome Research, LIIGH-UNAM (www.liigh.unam.mx) offers a tenure-track position at the level of Assistant Professor (Investigador Asociado C) to perform research, teaching and formation of human resources in the area of: “Genomics of Mendelian Diseases”

Applicants are expected to have a doctorate degree, postdoctoral experience related to the above mentioned area and a strong track record of peer-reviewed publications. Interested applicants must submit CV, email addresses of three references, and a three-page project to Dr. Rafael Palacios, Coordinator of LIIGH-UNAM (palacios@liigh.unam.mx) before June 21, 2019 ………………………………………………………………

Tenure Track position in Genomic Sciences

Laboratorio Internacional de Investigación sobre el Genoma Humano, UNAM Juriquilla, Querétaro, México

The International Laboratory for Human Genome Research, LIIGH-UNAM (www.liigh.unam.mx) offers a tenure-track position at the level of Assistant Professor (Investigador Asociado C) to perform research, teaching and formation of human resources in the area of: “Statistic Population Genomics and its Impact in Complex Diseases”

Applicants are expected to have a doctorate degree, postdoctoral experience related to the above mentioned area and a strong track record of peer-reviewed publications. Interested applicants must submit CV, email addresses of three references, and a three-page statement of research interests to Dr. Rafael Palacios, Coordinator of LIIGH-UNAM (palacios@liigh.unam.mx) before June 21, 2019