BOL: Related items

The Kingsley Lab

Fri, 03 Jun 2016 09:55:10 -0500

The Molecular Basis of Vertebrate Evolution. Naturally occurring species show spectacular differences in morphology, physiology, behavior, disease susceptibility, and life span. Although the genomes of many organisms have now been completely sequenced, Kingsley lab still know relatively little about the specific DNA sequence changes that underlie interesting species-specific traits. Kingsley lab laboratory is using a combination of genetic and genomic approaches to identify the detailed molecular mechanisms that control evolutionary change in vertebrates.

BBMap/BBTools package: Multipurpose tool designed for converting reads or other nucleotide data between different formats.

Jit — Mon, 13 Jun 2016 05:47:21 -0500

Reformatis a member of the BBMap/BBTools package. It is a multipurpose tool designed for converting reads or other nucleotide data between different formats. It supports, and can inter-convert:

fastq
fasta
fasta+qual
sam
scarf (an old Illumina format)
bam (if samtools is installed)
gzip
zip
ascii-33 (sanger)
ascii-64 (old Illumina)
paired files
interleaved files

It is multithreaded and can process data at over 500 megabytes per second, and can accept streams from standard in and write to standard out, allowing it to be easily dropped into the middle of a pipeline for format conversion. Reformat autodetects formats based on file extensions and content, making it very easy to use; and the autodetection can be overridden, allowing flexibility for people who don't like to follow naming conventions, or out-of-spec fastq files with qualities values like -17 or 120.

The program has been gradually expanded, and can now perform various other functions. None of these will break pairing, if the input is paired.

Quality trimming (either or both ends)
Quality filtering
Fixed-length trimming
Generation of histograms (base composition, quality, etc)
Subsampling (to a fraction of input reads, or an exact number of reads or bases)
Changing fasta line-wrapping length
Reverse-complementing (all reads or only read 2)
Adding /1 and /2 suffix to read names
GC-content filtering
Length-filtering
Testing for corrupted interleaved files

Reformat is compatible with any platform that supports Java 1.7 or higher. It also has a bash shellscript for simpler invocation. Typical usage examples:

Reformat fastq into fasta:
reformat.sh in=x.fq out=y.fa

Interleave paired reads:
reformat.sh in1=x1.fq in2=x2.fq out=y.fq

Note - you can actually use a shortcut if paired read files have the same name with a 1 and a 2. This is equivalent to the above command:
reformat.sh in=x#.fq out=y.fq

De-interleave reads:
reformat.sh in=x.fq out1=y1.fq out2=y2.fq

Verify that interleaving appears correct, assuming Illumina namimg conventions:
reformat.sh in=x.fq vint

Convert ASCII-33 to ASCII-64:
reformat.sh in=x.fq out=y.fq qin=33 qout=64

Quality-trim paired reads to Q10 on the left and right ends and discard reads shorter than 50bp after trimming:
reformat.sh in1=x1.fq in2=x2.fq out1=y1.fq out2=y2.fq outsingle=singletons.fq qtrim=rl trimq=10 minlength=50

Subsample 10% of the first 20000 pairs in an interleaved file:
reformat.sh in=x.fq out=y.fq reads=20000 samplerate=0.1 int=t
(in this case "int=t" overrides interleaving autodetection, to ensure reads are treated as pairs)

Pipe in a gzipped sam file and pipe out fasta:
reformat.sh in=stdin.sam.gz out=stdout.fa

Reverse-complement reads:
reformat.sh in=x.fq out=y.fq rcomp

For reformatting a file with very long sequences, Reformat will need more memory; just add the additional flag "-Xmx2g". For example, to change the line-wrapping length on the human genome (which has individual sequences over 200Mbp long) to 70 characters:
reformat.sh -Xmx2g in=HG19.fa.gz out=HG19_wrapped.fa.gz fastawrap=70

For additional functions, please run the shellscript with no arguments, or just read it with a text editor. If you have any questions, please post them in this thread.

For people using a non-bash terminal, you may need to type "bash reformat.sh" instead of just "reformat.sh".
For users of Windows or other platforms that do not support bash shellscripts, replace "reformat.sh" with "java -ea -Xmx200m /path/to/bbmap/current/ jgi.ReformatReads"
for example,
java -ea -Xmx200m C:\bbmap\current\ jgi.ReformatReads in=x.fq out=y.fa

Reformat can be downloaded with BBTools here:
https://sourceforge.net/projects/bbmap/

LoRMA: a tool for correcting sequencing errors in long reads such those produced by Pacific Biosciences sequencing machines

Jit — Wed, 15 Jun 2016 17:18:36 -0500

LoRMA is a tool for correcting sequencing errors in long reads such those produced by Pacific Biosciences sequencing machines.

Publication:

L. Salmela, R. Walve, E. Rivals, and E. Ukkonen: Accurate selfcorrection of errors in long reads using de Bruijn graphs. Accepted to RECOMB-Seq 2016.

Download:

Address of the bookmark: https://www.cs.helsinki.fi/u/lmsalmel/LoRMA/

clusterProfiler

Jit — Thu, 16 Jun 2016 18:57:03 -0500

statistical analysis and visulization of functional profiles for genes and gene clusters

Bioconductor version: Release (3.3)

This package implements methods to analyze and visualize functional profiles (GO and KEGG) of gene and gene clusters.

Author: Guangchuang Yu with contributions from Li-Gen Wang and Giovanni Dall'Olio.

Maintainer: Guangchuang Yu

Citation (from within R, enter citation("clusterProfiler")):

Yu G, Wang L, Han Y and He Q (2012). “clusterProfiler: an R package for comparing biological themes among gene clusters.” OMICS: A Journal of Integrative Biology, 16(5), pp. 284-287.
Installation

To install this package, start R and enter:

## try http:// if https:// URLs are not supported
source("https://bioconductor.org/biocLite.R")
biocLite("clusterProfiler")

https://www.bioconductor.org/packages/devel/bioc/vignettes/clusterProfiler/inst/doc/clusterProfiler.html

Address of the bookmark: https://www.bioconductor.org/packages/devel/bioc/vignettes/clusterProfiler/inst/doc/clusterProfiler.html

Cheatsheet for Linux !!

Jit — Wed, 22 Jun 2016 07:55:06 -0500

Linux Commands Cheat Sheet

    File System

    ls — list items in current directory

    ls -l — list items in current directory and show in long format to see perimissions, size, an modification date

    ls -a — list all items in current directory, including hidden files

    ls -F — list all items in current directory and show directories with a slash and executables with a star

    ls dir — list all items in directory dir

    cd dir — change directory to dir

    cd .. — go up one directory

    cd / — go to the root directory

    cd ~ — go to to your home directory

    cd - — go to the last directory you were just in

    pwd — show present working directory

    mkdir dir — make directory dir

    rm file — remove file

    rm -r dir — remove directory dir recursively

    cp file1 file2 — copy file1 to file2

    cp -r dir1 dir2 — copy directory dir1 to dir2 recursively

    mv file1 file2 — move (rename) file1 to file2

    ln -s file link — create symbolic link to file

    touch file — create or update file

    cat file — output the contents of file

    less file — view file with page navigation

    head file — output the first 10 lines of file

    tail file — output the last 10 lines of file

    tail -f file — output the contents of file as it grows, starting with the last 10 lines

    vim file — edit file

    alias name 'command' — create an alias for a command
    System

    shutdown — shut down machine

    reboot — restart machine

    date — show the current date and time

    whoami — who you are logged in as

    finger user — display information about user

    man command — show the manual for command

    df — show disk usage

    du — show directory space usage

    free — show memory and swap usage

    whereis app — show possible locations of app

    which app — show which app will be run by default
    Process Management

    ps — display your currently active processes

    top — display all running processes

    kill pid — kill process id pid

    kill -9 pid — force kill process id pid
    Permissions

    ls -l — list items in current directory and show permissions

    chmod ugo file — change permissions of file to ugo - u is the user's permissions, g is the group's permissions, and o is everyone else's permissions. The values of u, g, and o can be any number between 0 and 7.

    7 — full permissions

    6 — read and write only

    5 — read and execute only

    4 — read only

    3 — write and execute only

    2 — write only

    1 — execute only

    0 — no permissions

    chmod 600 file — you can read and write - good for files

    chmod 700 file — you can read, write, and execute - good for scripts

    chmod 644 file — you can read and write, and everyone else can only read - good for web pages

    chmod 755 file — you can read, write, and execute, and everyone else can read and execute - good for programs that you want to share
    Networking

    wget file — download a file

    curl file — download a file

    scp user@host:file dir — secure copy a file from remote server to the dir directory on your machine

    scp file user@host:dir — secure copy a file from your machine to the dir directory on a remote server

    scp -r user@host:dir dir — secure copy the directory dir from remote server to the directory dir on your machine

    ssh user@host — connect to host as user

    ssh -p port user@host — connect to host on port as user

    ssh-copy-id user@host — add your key to host for user to enable a keyed or passwordless login

    ping host — ping host and output results

    whois domain — get information for domain

    dig domain — get DNS information for domain

    dig -x host — reverse lookup host

    lsof -i tcp:1337 — list all processes running on port 1337
    Searching

    grep pattern files — search for pattern in files

    grep -r pattern dir — search recursively for pattern in dir

    grep -rn pattern dir — search recursively for pattern in dir and show the line number found

    grep -r pattern dir --include='*.ext — search recursively for pattern in dir and only search in files with .ext extension

    command | grep pattern — search for pattern in the output of command

    find file — find all instances of file in real system

    locate file — find all instances of file using indexed database built from the updatedb command. Much faster than find

    sed -i 's/day/night/g' file — find all occurrences of day in a file and replace them with night - s means substitude and g means global - sed also supports regular expressions
    Compression

    tar cf file.tar files — create a tar named file.tar containing files

    tar xf file.tar — extract the files from file.tar

    tar czf file.tar.gz files — create a tar with Gzip compression

    tar xzf file.tar.gz — extract a tar using Gzip

    gzip file — compresses file and renames it to file.gz

    gzip -d file.gz — decompresses file.gz back to file
    Shortcuts

    ctrl+a — move cursor to beginning of line

    ctrl+f — move cursor to end of line

    alt+f — move cursor forward 1 word

    alt+b — move cursor backward 1 word

Kraken: ultrafast metagenomic sequence classification using exact alignments

Jit — Mon, 27 Jun 2016 11:01:44 -0500

Kraken is an ultrafast and highly accurate program for assigning taxonomic labels to metagenomic DNA sequences. Previous programs designed for this task have been relatively slow and computationally expensive, forcing researchers to use faster abundance estimation programs, which only classify small subsets of metagenomic data. Using exact alignment of k-mers, Kraken achieves classification accuracy comparable to the fastest BLAST program. In its fastest mode, Kraken classifies 100 base pair reads at a rate of over 4.1 million reads per minute, 909 times faster than Megablast and 11 times faster than the abundance estimation program MetaPhlAn. Kraken is available at http://ccb.jhu.edu/software/kraken/.

Krona

https://sourceforge.net/p/krona/home/krona/

Address of the bookmark: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4053813/

Machine Learning !!!

Gudiya Pal — Fri, 01 Jul 2016 12:57:12 -0500

In machine learning, computers apply statistical learning techniques to automatically identify patterns in data. These techniques can be used to make highly accurate predictions.

Keep scrolling. Using a data set about homes, we will create a machine learning model to distinguish homes in New York from homes in San Francisco.

Address of the bookmark: http://www.r2d3.us/visual-intro-to-machine-learning-part-1/

Fancy Oneliner for Bioinformatics !!

Poonam Mahapatra — Thu, 07 Jul 2016 12:05:50 -0500

This webpage lists some of the one-liners that we frequently use in metagenomic analyses. You can click on the following links to browse through different topics. You can copy/paste the commands as they are in your terminal screen, provided you follow the same naming conventions and folder structures as we have. We are sharing these codes with the intention that if they are useful and help you in your analyses, then we will be appropriately credited as considerable effort has been put into devising them.

Address of the bookmark: http://userweb.eng.gla.ac.uk/umer.ijaz/bioinformatics/oneliners.html

BINC exam preparation tips !!

Jit — Fri, 15 Jul 2016 20:53:01 -0500

How to prepare for BINC (BioInformatics National Certification) exam? What are the expected questions?

These are just a scant few of the common questions asked by bioinformatics students as they ready themselves for the next exam sitting. If you read the entire Syllabus (and I know that everyone does), you will see a section devoted to study and exam techniques. The section discusses such broad concepts as motivation, scheduling, and retention. Upon reading this section, however, I find the "hints" to be too general. Much of the advice boils down to read, study, understand, and memorize the material. The techniques mentioned apply to everyone and thus the overall advice ends up as a broad overview of the learning process.

The idea behind this article is to give students ideas on different approaches and techniques in the preparation for exams. By providing various ways to prepare for the exam process, fascinated readers may gain some additional insight to help complement their studying methodology. There are, of course, many common themes expressed in this small empirical sample of students' study habits. The idea of note cards, memorization, and problem solving are frequently mentioned by all students. No matter what technique a candidate uses, it always takes a significant amount of time and personal resources to successfully complete the examination process.

1 Explain it in your own word

Your teacher or lecturer can explain something to you, you can learn it from a text book, your friends can study with you, even your own notes can explain it to you but all these explanations are of little use if, by the end, you can’t explain what you have learned to yourself. The BINC exam looking for ability to write and explain the concept in your own word. You, therefore, need to illustrate in an exam to get top exam results, then you won’t be happy with your end exam result. So don’t just memorise and tick off the list – make sure you understand your theory.

2 Be an examiner yourself

Of course, depending on what you’re studying, it may be quite difficult to get into a position to understand a concept, theory or other information you need to learn. Ask ‘stupid’ question to yourself and train yourself for the worst! Embrace your curiosity, for as William Arthur Ward said: “Curiosity is the wick in the candle of learning.” Doing so will allow you to fill in the blanks and better prepare you for exams.

3 Quiz yourself

Once you feel you understand topic, it is important to test yourself regularly. Try yourself to replicate exam conditions as much as possible: turn your phone off, don’t talk, time yourself etc. You can set yourself a study quiz or practice exam questions and, so long as you approach it with the right mindset, you can get a very good idea of how much you know. You gain a greater insight into where you stand in relation to what you’ve studied so far.

4 Online study

Keeping the fact that, bioinformatics is ever changing subject, you might need to update yourself on timely basis. Don’t feel obliged to just sit in front of a book with a highlighter; there are many different ways to improve your bioinformatics knowledge. Login and check almost all web servers and keep yourself updated, like how many genomes sequenced, sizes, techniques used, software names etc.

5 Study plan

In order to achieve exam success, you need to know what you want to achieve and focus on. That’s why it is extremely important to set your Study Goals now and outline to yourself what you need to do. With your study goals in mind, you properly need to attention all subjects. It should be broad enough to allow you to add and change aspects but concise enough so you know you’re covering each subject/topic as best you can at this point.

MEGAN6

Neel — Mon, 25 Jul 2016 05:45:22 -0500

Microbiome analysis using a single application

MEGAN6 is a comprehensive toolbox for interactively analyzing microbiome data. All the interactive tools you need in one application.

Taxonomic analysis using the NCBI taxonomy or a customized taxonomy such as SILVA
Functional analysis using InterPro2GO, SEED, eggNOG or KEGG
Bar charts, word clouds, Voronoi tree maps and many other charts
PCoA, clustering and networks
Supports metadata
MEGAN parses many different types of input

Why use MEGAN6?

The software is:

Easy to use. MEGAN6 is a single application and all features are available through menus, toolbars and graphics. No scripting skills required.
Powerful. MEGAN6 allows you to work with hundreds of samples containing hundreds of millions of sequencing reads. Blast-like analysis can be performed using DIAMOND.
Comprehensive. MEGAN6 offers a large range of analysis tools, and is under active development.

Address of the bookmark: https://ab.inf.uni-tuebingen.de/software/megan6