In comparative genome analysis work, we usually compare more than two genomes and looks for syntenic regions amongst them. In my research I used Evolution Highway (RH) http://eh-demo.ncsa.uiuc.edu/, which is a collaborative project designed to provide a visual means for simultaneously comparing genomes of multiple amniote species. The tool removes the burden of manually aligning these maps and allows cognitive skills to be used toward something more valuable than preparation and transformation of data. In addition to EH, attractive Circos (http://circos.ca/) is also very popular for this kind of analysis.

The EH is available online, and can be easily access and use, whereas Circos installation is not entirely straightforward. One of the most difficult parts of the installation involves installing the GD library. Since there weren't good instructions for installing this library on the internet I decided to post instructions here in case they are useful to anyone else.

Following are the steps to install GD modules in Mac OS

1. Setup

Create a folder for the files:

$ mkdir -p /SourceCache
$ cd /SourceCache

Get and unpack the required Jpeg-6b and GD libraries:
Download Jpeg-6b (http://code.google.com/p/google-desktop-for-linux-mirror/downloads/detail?name=jpeg-6b.tar.gz&can=2&q)
Download GD (http://search.cpan.org/~lds/GD-2.46/)

Place the "tar.gz" files in "/SourceCache" and double click to unpack.

2. Install libjpeg

Copy the "config.sub" and "config.guess" files to "/SourceCache". Note that your "config.sub" and ""config.guess" files may be in a slightly different location. The commands below show where they were on my machine:

$ cd /SourceCache/jpeg-6b/src
$ cp /usr/share/libtool/config/config.sub .
$ cp /usr/share/libtool/config/config.guess .

Configure libjpeg as follows. Note that this was installed on a 64 bit machine. However, this method may configure it in a 32 bit format. This may not be the best way to configure the installation but it works.

$ .configure --enable-shared
$ make

Check to see if the following directories exist on your machine. Create the missing directories in the following manner:

$ mkdir -p /usr/local/include
$ mkdir -p /usr/local/bin
$ mkdir -p /usr/local/lib
$ mkdir -p /usr/local/man/man1

Finish making and installing libjpeg:

$ make install

3. Install GD

$ cd /SourceCache/GD-2.46/GD/
$ perl Makefile.PL
$ make
$ make test (optional)
$ make html (optional)
$ make install

Other way for Mac OS
The easiest way to get a lot of these is with a program called Fink, which is similar in nature to the CPAN installer, but installs common GNU utilities. Fink is available from <http://sourceforge.net/projects/fink/>.

Follow the instructions for setting up Fink. Once it's installed, you'll want to run the following as root: fink install gd

It will prompt you for a number of dependencies, type 'y' and hit enter to install all of the dependencies. Then watch it work.

To prevent creating conflicts with the software that Apple installs by default, Fink creates its own directory tree at /sw where it installs most of the software that it installs. This means your libraries and headers for libgd will be at /sw/lib and /sw/include instead of /usr/lib and /usr/local/include. Because of these changed locations for the libraries, the Perl GD module will not install directly via CPAN, because it looks for the specific paths instead of getting them from your environment. But there's a way around that :-)

Instead of typing "install GD" at the cpan> prompt, type look GD. This should go through the motions of downloading the latest version of the GD module, then it will open a shell and drop you into the build directory. Apply below patch to the Makefile.PL file (save the patch into a file and use the command patch < patchfile.)

Then, run these commands to finish the installation of the GD module:

perl Makefile.PL
make
make test
make install
And don't forget to run exit to get back to CPAN.

Install on MS Window, using PPM

C:\Documents and Settings\Owner>ppm
PPM interactive shell (2.2.0) - type 'help' for available commands.
PPM> install GD
Install package 'GD?' (y/N): y
Installing package 'GD'...
Downloading http://ppm.ActiveState.com/PPMPackages/5.6plus/MSW. ...
Installing C:\Perl\site\lib\auto\GD\GD.bs
Installing C:\Perl\site\lib\auto\GD\GD.dll
Installing C:\Perl\site\lib\auto\GD\GD.exp
Installing C:\Perl\site\lib\auto\GD\GD.lib
Installing C:\Perl\html\site\lib\GD.html
Installing C:\Perl\site\lib\GD.pm
Installing C:\Perl\site\lib\qd.pl
Installing C:\Perl\site\lib\auto\GD\autosplit.ix
PPM>


If you can't install it from ppm. You can download it:
http://ppm.ActiveState.com/PPMPackages/5.6plus/MSW.


BTW,All Perl 5.6.1 Modules are located at:

http://ppm.ActiveState.com/PPMPackages/5.6plus/MSW.

Install the Perl GD Module on Linux

$ sudo perl -MCPAN -e shell

Since it was the first time I had run this command on this particular machine I had to answer a lot of questions but simply selected the defaults for everything as this usually works for me. Once in the CPAN shell I entered

$ install Bundle::CPAN

and selected all of the defaults again. Once the CPAN bundle had finished installing I tried to install GD::Graph by typing

$ install GD::Graph

but it failed with hundreds of errors – the first of which was

GD.xs:7:16: error: gd.h: No such file or directory

This was fixed with the following apt-get command (in the bash shell)

$ sudo apt-get install libgd2-xpm-dev

back in the CPAN shell I still couldn’t get GD::Graph to build and I guessed this was because of some left over files from the failed build. I don’t know the command to clean things up inside the CPAN shell and am too lazy to read the docs so I simply went into the .cpan/build directory in my home directory and deleted anything that started with GD – eg

$ rm -rf GD-2.35-HC_vkB

$ rm -rf GDGraph-1.44-Evfibe

and so on. Those strings at the end (VkB and so on) look random so they might be different on your machine. Then I went back into the CPAN shell and ran

$ install GD::Graph

There were a few dependencies which the script fetched and installed for me but everything worked smoothly.

Manual and other Perl Module instalation are mentioned in my previous blog @ http://bioinformaticsonline.com/blog/view/710/how-to-install-perl-modules-manually-using-cpan-command-and-other-quick-ways

In our day to day research activity, we need to securely copy our data from several to local computer and visa-versa. I am jotting down some of the commonly used SCP command for your future help. Hope you all will like it

What is Secure Copy?

scp allows files to be copied to, from, or between different hosts. It uses ssh for data transfer and provides the same authentication and same level of security as ssh.

Examples

Copy the file "gene.txt" from a remote host to the local host

    $ scp your_username@remotehost.edu:gene.txt /some/local/directory

Copy the file "foobar.txt" from the local host to a remote host

    $ scp gene.txt your_username@remotehost.edu:/some/remote/directory

Copy the directory "chromosome" from the local host to a remote host's directory "bar"

    $ scp -r chromosome your_username@remotehost.edu:/some/remote/directory/bar

Copy the file "gene.txt" from remote host "rh1.edu" to remote host "rh2.edu"

    $ scp your_username@rh1.edu:/some/remote/directory/gene.txt \
    your_username@rh2.edu:/some/remote/directory/

Copying the files "gene.txt" and "cancer.txt" from the local host to your home directory on the remote host

    $ scp gene.txt cancer.txt your_username@remotehost.edu:~

Copy the file "gene.txt" from the local host to a remote host using port 2264

    $ scp -P 2264 gene.txt your_username@remotehost.edu:/some/remote/directory

Copy multiple files from the remote host to your current directory on the local host

    $ scp your_username@remotehost.edu:/some/remote/directory/\{a,b,c\} .

    $ scp your_username@remotehost.edu:~/\{gene.txt,cancer.txt\} .

scp Performance

By default scp uses the Triple-DES cipher to encrypt the data being sent. Using the Blowfish cipher has been shown to increase speed. This can be done by using option -c blowfish in the command line.

    $ scp -c blowfish some_file your_username@remotehost.edu:~

It is often suggested that the -C option for compression should also be used to increase speed. The effect of compression, however, will only significantly increase speed if your connection is very slow. Otherwise it may just be adding extra burden to the CPU. An example of using blowfish and compression:

    $ scp -c blowfish -C local_file your_username@remotehost.edu:~

Use the Terminal or an Anaconda Prompt for the following steps.

1. To create an environment:

   conda create --name myenv
   

   NOTE: Replace myenv with the environment name.

2. When conda asks you to proceed, type y:

   proceed ([y]/n)?
   

This creates the myenv environment in /envs/. This environment uses the same version of Python that you are currently using, because you did not specify a version.

To create an environment with a specific version of Python:

conda create -n myenv python=3.4

To create an environment with a specific package:

conda create -n myenv scipy

OR:

conda create -n myenv python
conda install -n myenv scipy

To create an environment with a specific version of a package:

conda create -n myenv scipy=0.15.0

OR:

conda create -n myenv python
conda install -n myenv scipy=0.15.0

To create an environment with a specific version of Python and multiple packages:

conda create -n myenv python=3.4 scipy=0.15.0 astroid babel

TIP: Install all the programs that you want in this environment at the same time. Installing 1 program at a time can lead to dependency conflicts.

To automatically install pip or another program every time a new environment is created, add the default programs to the create_default_packages section of your .condarc configuration file. The default packages are installed every time you create a new environment. If you do not want the default packages installed in a particular environment, use the --no-default-packages flag:

conda create --no-default-packages -n myenv python

TIP: You can add much more to the conda create command. For details, run conda create --help.

➜ redundans git:(master) ✗ conda create --name py27 python=2.7
Solving environment: done

==> WARNING: A newer version of conda exists. <==
current version: 4.5.0
latest version: 4.5.2

Please update conda by running

$ conda update -n base conda

## Package Plan ##

environment location: /home/urbe/anaconda3/envs/py27

added / updated specs:
- python=2.7

The following packages will be downloaded:

package | build
---------------------------|-----------------
wheel-0.31.0 | py27_0 61 KB
python-2.7.15 | h1571d57_0 12.1 MB
certifi-2018.4.16 | py27_0 142 KB
sqlite-3.23.1 | he433501_0 1.5 MB
setuptools-39.1.0 | py27_0 582 KB
openssl-1.0.2o | h20670df_0 3.4 MB
pip-10.0.1 | py27_0 1.7 MB
ca-certificates-2018.03.07 | 0 124 KB
------------------------------------------------------------
Total: 19.6 MB

The following NEW packages will be INSTALLED:

ca-certificates: 2018.03.07-0
certifi: 2018.4.16-py27_0
libedit: 3.1-heed3624_0
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 7.2.0-hdf63c60_3
libstdcxx-ng: 7.2.0-hdf63c60_3
ncurses: 6.0-h9df7e31_2
openssl: 1.0.2o-h20670df_0
pip: 10.0.1-py27_0
python: 2.7.15-h1571d57_0
readline: 7.0-ha6073c6_4
setuptools: 39.1.0-py27_0
sqlite: 3.23.1-he433501_0
tk: 8.6.7-hc745277_3
wheel: 0.31.0-py27_0
zlib: 1.2.11-ha838bed_2

Proceed ([y]/n)? y

Downloading and Extracting Packages
wheel 0.31.0: #################################################################################################################################################################################################### | 100%
python 2.7.15: ################################################################################################################################################################################################### | 100%
certifi 2018.4.16: ############################################################################################################################################################################################### | 100%
sqlite 3.23.1: ################################################################################################################################################################################################### | 100%
setuptools 39.1.0: ############################################################################################################################################################################################### | 100%
openssl 1.0.2o: ################################################################################################################################################################################################## | 100%
pip 10.0.1: ###################################################################################################################################################################################################### | 100%
ca-certificates 2018.03.07: ###################################################################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate py27
#
# To deactivate an active environment, use:
# > source deactivate
#

➜ redundans git:(master) ✗ source activate py27

https://www.forbes.com/sites/jasonevangelho/2018/07/23/5-reasons-you-should-switch-from-windows-to-linux-right-now/#70c74923777b

The following table shows SLURM commands on the SOE cluster.

For more information, run man on the commands above. See some examples below.

1. Info about the partitions and nodes
List all the partitions available to you and the nodes therein:

sinfo

Nodes in state idle can accept new jobs.

Show a partition configuratuin, for example, SOE_main

scontrol show partition=SOE_main

Show current info about a specific node:

scontrol show node=<nodename>

You can also specify a group of nodes in the command above. For example, if your MPI job is running across soenode05,06,35,36, you can execute the command below to get the info on the nodes you are interested in:

scontrol show node=soenode[05-06,35-36]

An informative parameter in the output to look at would be CPULoad. It allows you to see how your application utilizes the CPUs on the running nodes.

2. Submit scripts
The header in a submit script specifies job name, partition (queue), time limit, memory allocation, number of nodes, number of cores, and files to collect standard output and error at run time, for example

#!/bin/bash

#SBATCH --job-name=OMP_run     # job name, "OMP_run"
#SBATCH --partition=SOE_main   # partition (queue)
#SBATCH -t 0-2:00              # time limit: (D-HH:MM) 
#SBATCH --mem=32000            # memory per node in MB 
#SBATCH --nodes=1              # number of nodes
#SBATCH --ntasks-per-node=16   # number of cores
#SBATCH --output=slurm.out     # file to collect standard output
#SBATCH --error=slurm.err      # file to collect standard errors

If the time limit is not specified in the submit script, SLURM will assign the default run time, 3 days. This means the job will be terminated by SLURM in 72 hrs. The maximum allowed run time is two weeks, 14-0:00.
If the memory limit is not requested, SLURM will assign the default 16 GB. The maximum allowed memory per node is 128 GB. To see how much RAM per node your job is using, you can run commands sacct or sstat to query MaxRSS for the job on the node - see examples below.
Depending on a type of application you need to run, the submit script may contain commands to create a temporary space on a computational node - see the discussion about using the file systems on the cluster.
Then it sets the environment specific to the application and starts the application on one or multiple nodes - see sbatch sample scripts in directory /usr/local/Samples on soemaster1.hpc.rutgers.edu.
You can submit your job to the cluster with sbatch command:

sbatch myscript.sh

3. Query job information
List all currently submitted jobs in running and pending states for a user:

squeue -u <username>

Command squeue can be run with format options to expose specific information, for example, when pending job #706 is scheduled to start running:

squeue -j 706 --format="%S"

START_TIME
2015-04-30T09:54:32

More info can be shown by placing additional format options, for example:

squeue -j 706 --format="%i %P %j %u %T %l %C %S"

JOBID PARTITION   NAME    USER STATE   TIMELIMIT  CPUS START_TIME
706   SOE_main  Par_job_3 mike PENDING 3-00:00:00 64   2015-04-30T09:54:32

To see when all the jobs, pending in the queue, are scheduled to start:

squeue --start 

List all running and completed jobs for a user

sqlog -u <username>

or

sqlog -j <JobID>

The following appreviations are used for the job states:

       CA   CANCELLED      Job was cancelled.

       CD   COMPLETED      Job completed normally.

       CG   COMPLETING     Job is in the process of completing.

       F    FAILED         Job termined abnormally.

       NF   NODE_FAIL      Job terminated due to node failure.

       PD   PENDING        Job is pending allocation.

       R    RUNNING        Job currently has an allocation.

       S    SUSPENDED      Job is suspended.

       TO   TIMEOUT        Job terminated upon reaching its time limit.

You can specify the fields you would like to see in the output of sqlog:

sqlog --format=list

The command below, for example, provides Job ID, user name, exit state, start date-time, and end date-time for job #2831:

sqlog -j 2831 --format=jid,user,state,start,end

List status info for a currently running job:

sstat -j <jobid>

A formatted output can be used to gain only a specific info, for example, the maximum resident RAM usage on a node:

sstat --format="JobID,MaxRSS" -j <jobid>

To get statistics on completed jobs by jobID:

sacct --format="JobID,JobName,MaxRSS,Elapsed" -j <jobid>

To view the same information for all jobs of a user:

sacct --format="JobID,JobName,MaxRSS,Elapsed" -u <username>

To print a list of fields that can be specified with the --format option:

sacct --helpformat

For example, to get Job ID, Job name, Exit state, start date-time, and end date-time for job #2831:

sacct -j 2831 --format="JobID,JobName,State,Start,End"

Another useful command to gain information about a running job is scontrol:

scontrol show job=<jobid>

4. Cancel a job
To cancel one job:

scancel <jobid>

To cancel one job and delete the TMP directory created by the submit script on a node:

sdel <jobid>

To cancel all the jobs for a user:

scancel -u <username>

To cancel one or more jobs by name:

scancel --name <myJobName>

Address of the bookmark: http://bioinformatics.oxfordjournals.org/content/23/14/1713.full

Spines

http://www.broadinstitute.org/science/programs/genome-biology/spines

Address of the bookmark: http://www.broadinstitute.org/science/programs/genome-biology/spines

https://github.com/pditommaso/awesome-pipeline

Address of the bookmark: https://github.com/pditommaso/awesome-pipeline

LPI scores are inversely proportional to the phylogenetic distance between database match sequences and the query genome. These scores are useful not only for large-scalede novo predictions of horizontally transferred proteins, but can also serve as an independent quality control test for potential horizontal transfer candidates identified by alternative methods, especially those based on nucleic acid signatures. Candidates having high LPI scores are unlikely to have been horizontally transferred, since they are highly conserved among closely related organisms.

One unique and powerful feature of the DarkHorse HGT Candidate database is the opportunity to explore the phylogenetic background of potential HGT donors as well as recipients. The breadth of the database allows not only query sequences, but also their database match partners to be evaluated for sequence similarity or novelty compared to taxonomically related organisms.

DarkHorse is configurable for varying degrees of phylogenetic granularity and protein sequence conservation. Users should consult the references cited below for a complete explanation of parameter selection and result interpretation. A brief tutorial page is also available on-line.

Address of the bookmark: http://darkhorse.ucsd.edu/download.html