bwa aln hs37m.fa r1.fq > r1.sai && bwa aln hs37m.fa r2.fq > r2.sai \
&& bwa sampe hs37m r1.sai r2.sai r1.fq r2.fq > r12.bwa.sam

bwa bwasw ../index/bwa/hs37m.fa r12.fq > r12.bwasw.sam

gsnap -A sam -d hs37m r1.fq r2.fq > r12.gsnap.sam

novoalign -r Random -o SAM -f r1.fq r2.fq -i 500 50 -d hs37m-k14s3.novo > r12.novo.sam

smalt map -f samsoft -i 650 -o r12.smalt-k20s13.sam hs37m-k20s13 r1.fq r2.fq

stampy.py -g hs37m -h hs37m -o r12.stampy.sam -M r1.fq,r2.fq

soap -D hs37m.fa.index -a r1.fq -b r2.fq -l 32 -g 3 -u dummy -2 dummy -o r12.soap

java -Xmx8g -jar GenomeAnalysisTK.jar \
-T VariantsToTable \
-R reference.fa \
-V reference_genomes_GT.vcf \
-F CHROM -F POS -F REF -F ALT -GF GT \
-o reference_genomes_GT.table
multiple_sample.vcf should also be converted to multiple_sample_GT.table using this approach.

In this tutorial series article, I will explore features and packages of python which are widely used in the big data, NGS, and bioinformatics. I will also walk through a real biological example which shows NGS data processing with the help of python packages and programming.

Python has a couple of points to recommend it to biologists and scientists specifically:

• It's widely used in the scientific community
• It has a couple of very well designed libraries for doing complex scientific computing (although we won't encounter them in this book)
• It lend itself well to being integrated with other, existing tools
• It has features which make it easy to manipulate strings of characters (for example, strings of DNA bases and protein amino acid residues, which we as biologists are particularly fond of)

In general, following are some of the important features of python which makes it a perfect fit for rapid application development.

• Python is interpreted language so the program does not need to be compiled. Interpreter parses the program code and generates the output.
• Python is dynamically typed, so the variables types are defined automatically.
• Python is strongly typed. So the developers need to cast the type manually.
• Less code and more use makes it more acceptable.
• Python is portable, extendable and scalable.

There are two major Python versions, Python 2 and Python 3. Python 2 and 3 are quite different. This tutorial uses Python 3, because it more semantically correct and supports newer features.

I will post tutorial on daily basis on this page. Check the sub-pages on right side.

Use the Terminal or an Anaconda Prompt for the following steps.

1. To create an environment:

   conda create --name myenv
   

   NOTE: Replace myenv with the environment name.

2. When conda asks you to proceed, type y:

   proceed ([y]/n)?
   

This creates the myenv environment in /envs/. This environment uses the same version of Python that you are currently using, because you did not specify a version.

To create an environment with a specific version of Python:

conda create -n myenv python=3.4

To create an environment with a specific package:

conda create -n myenv scipy

OR:

conda create -n myenv python
conda install -n myenv scipy

To create an environment with a specific version of a package:

conda create -n myenv scipy=0.15.0

OR:

conda create -n myenv python
conda install -n myenv scipy=0.15.0

To create an environment with a specific version of Python and multiple packages:

conda create -n myenv python=3.4 scipy=0.15.0 astroid babel

TIP: Install all the programs that you want in this environment at the same time. Installing 1 program at a time can lead to dependency conflicts.

To automatically install pip or another program every time a new environment is created, add the default programs to the create_default_packages section of your .condarc configuration file. The default packages are installed every time you create a new environment. If you do not want the default packages installed in a particular environment, use the --no-default-packages flag:

conda create --no-default-packages -n myenv python

TIP: You can add much more to the conda create command. For details, run conda create --help.

➜ redundans git:(master) ✗ conda create --name py27 python=2.7
Solving environment: done

==> WARNING: A newer version of conda exists. <==
current version: 4.5.0
latest version: 4.5.2

Please update conda by running

$ conda update -n base conda

## Package Plan ##

environment location: /home/urbe/anaconda3/envs/py27

added / updated specs:
- python=2.7

The following packages will be downloaded:

package | build
---------------------------|-----------------
wheel-0.31.0 | py27_0 61 KB
python-2.7.15 | h1571d57_0 12.1 MB
certifi-2018.4.16 | py27_0 142 KB
sqlite-3.23.1 | he433501_0 1.5 MB
setuptools-39.1.0 | py27_0 582 KB
openssl-1.0.2o | h20670df_0 3.4 MB
pip-10.0.1 | py27_0 1.7 MB
ca-certificates-2018.03.07 | 0 124 KB
------------------------------------------------------------
Total: 19.6 MB

The following NEW packages will be INSTALLED:

ca-certificates: 2018.03.07-0
certifi: 2018.4.16-py27_0
libedit: 3.1-heed3624_0
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 7.2.0-hdf63c60_3
libstdcxx-ng: 7.2.0-hdf63c60_3
ncurses: 6.0-h9df7e31_2
openssl: 1.0.2o-h20670df_0
pip: 10.0.1-py27_0
python: 2.7.15-h1571d57_0
readline: 7.0-ha6073c6_4
setuptools: 39.1.0-py27_0
sqlite: 3.23.1-he433501_0
tk: 8.6.7-hc745277_3
wheel: 0.31.0-py27_0
zlib: 1.2.11-ha838bed_2

Proceed ([y]/n)? y

Downloading and Extracting Packages
wheel 0.31.0: #################################################################################################################################################################################################### | 100%
python 2.7.15: ################################################################################################################################################################################################### | 100%
certifi 2018.4.16: ############################################################################################################################################################################################### | 100%
sqlite 3.23.1: ################################################################################################################################################################################################### | 100%
setuptools 39.1.0: ############################################################################################################################################################################################### | 100%
openssl 1.0.2o: ################################################################################################################################################################################################## | 100%
pip 10.0.1: ###################################################################################################################################################################################################### | 100%
ca-certificates 2018.03.07: ###################################################################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate py27
#
# To deactivate an active environment, use:
# > source deactivate
#

➜ redundans git:(master) ✗ source activate py27

##################################################
[Mon May 7 11:29:18 2018] Reduction...
#file name genome size contigs heterozygous size [%] heterozygous contigs [%] identity [%] possible joins homozygous size [%] homozygous contigs [%]
/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
test/run1/contigs.fa 163897 245 66377 40.50 221 90.20 94.854 0 97520 59.50 24 9.80

##################################################
[Mon May 7 11:29:29 2018] Estimating parameters of libraries...
Aligning 19504 mates per library...
Insert size statistics Mates orientation stats
FastQ files read length median mean stdev FF FR RF RR
test/5000_1.fq.gz test/5000_2.fq.gz 50 4998 4990.20 721.47 0 4674 0 0
test/600_1.fq.gz test/600_2.fq.gz 100 599 598.63 47.68 0 10000 0 0

##################################################
[Mon May 7 11:29:29 2018] Scaffolding...
iteration 1.1: test/run1/contigs.reduced.fa 24 97520 39.355 17 94157 7321 2195 0 29603
19505 pairs. 17302 passed filtering [88.71%]. 1627 in different contigs [8.34%].
1526 pairs. 558 in different contigs [36.57%].
iteration 1.2: test/run1/_sspace.1.1.fa 3 97626 39.344 3 97626 87536 6063 821 87536
19505 pairs. 17607 passed filtering [90.27%]. 182 in different contigs [0.93%].
1077 pairs. 124 in different contigs [11.51%].
iteration 2.1: test/run1/_sspace.1.2.fa 3 97626 39.344 3 97626 87536 6063 821 87536
19505 pairs. 15112 passed filtering [77.48%]. 1295 in different contigs [6.64%].
3417 pairs. 396 in different contigs [11.59%].
iteration 2.2: test/run1/_sspace.2.1.fa 1 99133 39.344 1 99133 99133 99133 2328 99133
19505 pairs. 15152 passed filtering [77.68%]. 0 in different contigs [0.00%].
3398 pairs. 0 in different contigs [0.00%].

##################################################
[Mon May 7 11:29:34 2018] Gap closing...
iteration 1.1: test/run1/scaffolds.fa 1 99133 39.344 1 99133 99133 99133 2328 99133

##################################################
[Mon May 7 11:29:35 2018] Final reduction...
#file name genome size contigs heterozygous size [%] heterozygous contigs [%] identity [%] possible joins homozygous size [%] homozygous contigs [%]
[WARNING] Nothing reduced!
test/run1/scaffolds.filled.fa 99390 1 0 0.00 0 0.00 0.000 0 99390 100.00 1 100.00

##################################################
[Mon May 7 11:29:35 2018] Reporting statistics...
#fname contigs bases GC [%] contigs >1kb bases in contigs >1kb N50 N90 Ns longest
test/contigs.fa 245 163897 40.298 24 117391 3975 233 0 29603
test/run1/contigs.fa 245 163897 40.298 24 117391 3975 233 0 29603
test/run1/contigs.reduced.fa 24 97520 39.355 17 94157 7321 2195 0 29603
test/run1/_sspace.1.1.fa 3 97626 39.344 3 97626 87536 6063 821 87536
test/run1/_sspace.1.2.fa 3 97626 39.344 3 97626 87536 6063 821 87536
test/run1/_sspace.2.1.fa 1 99133 39.344 1 99133 99133 99133 2328 99133
test/run1/_sspace.2.2.fa 1 99133 39.344 1 99133 99133 99133 2328 99133
test/run1/scaffolds.fa 1 99133 39.344 1 99133 99133 99133 2328 99133
test/run1/_gapcloser.1.1.fa 1 99390 39.689 1 99390 99390 99390 2 99390
test/run1/scaffolds.filled.fa 1 99390 39.689 1 99390 99390 99390 2 99390
test/run1/scaffolds.reduced.fa 1 99390 39.689 1 99390 99390 99390 2 99390

##################################################
[Mon May 7 11:29:35 2018] Cleaning-up...
#Time elapsed: 0:00:17.376924

3D-Dock Suite

Global rigid search: FFTShape complementarity and electrostatics

Re-scoring and clustering. Refinement of interface side-chains

3D-Garden

Global rigid search in ensamble

Shape complementarity and Lennard–Jones potential

Side chain and backbone dihedral refinement

DOT

Global rigid search: FFTShape complementarity, electrostatics and VDWNone

Escher NG

Global rigid searchShape complementarity, hydrogen bonds and electrostatic

Integrated in VEGA

GRAMM 

Global rigid search: FFT. smooth protein surface representation for soft docking

Shape complementarity and Lennard-Jones potential

Clustering of conformations

GRAMM-X 

Global rigid search: FFT. smooth protein surface representation for soft docking

Shape complementarity and Lennard-Jones potentialminimization and re-scoring with multiple filters

HEX

Global rigid search: Fourier correlation of spherical harmonics

Shape complementarity

HADDOCK

Global rigid searchElectrostatic ,VDW and desolvation energy termsMD simulated annealing refinement . Filtering based on external data. 

ICM

Global rigid search: Monte CarloEmpirical scoring function

Clustering and selection of conformations. Refinement of interface side-chains and re-scoring

MolFit 

Global rigid search: FFTShape complementarity

Clustering of good solutions, filtering using a priori information and small, local rigid rotations around selected conformations

PatchDock

Global rigid searchShape complementarity and atomic desolvation energy

Clustering of conformations

PyDock

Global rigid search:FFTShape complementarity

rescoring by binding electrostatics and desolvation energy

RosettaDock

Local rigid search: Monte Carlo with low and high resolution structure representation levels

Different scoring parameters for the different resolutions 

ZDOCK

Global rigid search: FFTShape complementarity, desolvation energy, and electrostatics.

Energy minimization and re-scoringFree for academics

Point to note:

The proper treatment of flexibility in protein–protein docking is still an active field of research. You first should analyzed your proteins in order to define their conformational space and then choose the most suitable method for your docking problem.

AutoDock

Stochastic (GA)

Flexible ligand and partially flexible target

ArgusLab

Systematic

Flexible ligandX-Score based

DOCK

Systematic (IC)

Flexible ligandDOCK 3.5 (force field)

eHITS

Systematic (RBD of fragments followed by reconstruction)Flexible ligand and partially flexible targetHiTS_Score (empirical)

FlexX

Systematic (IC)Flexible ligandFlexX SF (empirical)Commercial

FLIPDock

Stochastic (GA)Flexible ligand and flexible targetAUTODOCK (empirical)

FRED

Systematic (RBD)Flexible ligandChemScore, PLP, ScreenScore, ChemGauss (empirical/consensus)

GOLD

Stochastic (GA)

Flexible ligand and partially flexible targetGoldScore, ChemScore (empirical), ASP (knowledge based)

ICM

Stochastic (MC)

Flexible ligand and partially flexible targetICM SF (empirical)

ParDOCK

Stochastic (MC)

RigidBAPPL (empirical)

PLANTS

Stochastic (ACO)Flexible ligand and partially flexible target

CHEMPLP, PLP (empirical)

Surflex

Systematic (IC/MA)Flexible ligandHammerhead based (empirical)

Point to note:

Several studies have shown that the performance of most docking tools is highly dependent on the particular characteristics of both the binding site and the ligand to be investigated, and the determination which method would be more suitable in a specific context is difficult. We encouraged you to check several docking methods to determine which one(s) work best for your system.

Following are the list of tools commonly used to PPIs predictions:

Protein-Protein Interaction Sites

PPISP

A consensus neural network method for predicting protein-protein interaction sites

HOMCOS

A server to predict interacting protein pairs and interacting sites by homology modeling of complex structures

HotPOINT

Prediction of protein interfaces using an empirical model

ISIS

Prediction of interaction hotspots from sequence

KFC server

Automated decision-tree approach to predicting protein-protein interaction hot spots

meta-PPISP

A meta server for predicting protein-protein interaction sites. meta-PPISP is built on three individual web servers: cons-PPISP, PINUP, and Promate

ODA

Identification of optimal surface patches with the lowest docking desolvation energy values

PINUP

Protein binding site prediction with an empirical scoring function

Other Sites (DNA, RNA, Metals)

CHED 

Web server for predicting soft metal binding sites in proteins

DBD-Hunter

A knowledge-based method for the prediction of DNA-protein interactions

DISPLAR

Given the structure of a protein known to bind DNA, the method predicts residues that contact DNA using neural network method

iDBPs

Predicts DNA binding proteins for proteins with known 3D structure.

PFplus

Pockets Identification

CASTp

Pocket-Finder

PocketPicker