Shell history search command

Type history at a shell prompt:
$ history

It will display the list of all used commandline history with an serial number.

To search particular command, enter:
$ history | grep command-name
$ history | egrep -i 'scp|ssh|ftp'
Emacs Line-Edit Mode Command History Searching

To get previous command containing string, hit [CTRL]+[r] followed by search string:

(reverse-i-search):

To get previous command, hit [CTRL]+[p]. You can also use up arrow key.

CTRL-p

To get next command, hit [CTRL]+[n]. You can also use down arrow key.

CTRL-n

fc command

Apart from hostory command there are fc command to extract the command from history. The fc stands for either "find command" or "fix command.

For example list last 10 command, enter:
$ fc -l 10
To list commands 130 through 150, enter:
$ fc -l 130 150
To list all commands since the last command beginning with ssh, enter:
$ fc -l ssh
You can edit commands 1 through 5 using vi text editor, enter:
$ fc -e vi 1 5

Delete command history

The -c option causes the history list to be cleared by deleting all of the entries:
$ history -c

Screen is like a window manager for your console. It will allow you to keep multiple terminal sessions running and easily switch between them. It also protects you from disconnection, because the screen session doesn’t end when you get disconnected.

You’ll need to make sure that screen is installed on the server you are connecting to. If that server is Ubuntu or Debian, just use this command:

sudo apt-get install screen

Now you can start a new screen session by just typing screen at the command line. You’ll be shown some information about screen. Hit enter, and you’ll be at a normal prompt.

To disconnect (but leave the session running)

Hit Ctrl + A and then Ctrl + D in immediate succession. You will see the message [detached]

To reconnect to an already running session

screen -r

To reconnect to an existing session, or create a new one if none exists

screen -D -r

To create a new window inside of a running screen session

Hit Ctrl + A and then C in immediate succession. You will see a new prompt.

To switch from one screen window to another

Hit Ctrl + A and then Ctrl + A in immediate succession.

To list open screen windows

Hit Ctrl + A and then W in immediate succession

'du' – Check the size of a directory

$ du
This command ( du) gives you a list of directories that exist in the current working directory along with their sizes in kilobytes (default). The last line of the output gives you the total size of the current directory including its subdirectories.

$ du /home/jin1
The above command would give you the directory size of the directory /home/david

$ du -h
The same “du”command with some flag gives you a better output than the default one. The option '-h' stands for human readable format. Therefore, in order to print the sizes of the files / directories in your desire notation use this time suffixed with a 'k' if its kilobytes and 'M' if its Megabytes and 'G' if its Gigabytes.

$ du -ah
If you are interested in checking everything present in a folder use above mentioned command. It gives us not only the directories but also all the files that are present in the current directory. The “-a” flag displays the filenames along with the directory names in the output.

$ du -c
This gives you a grand total as the last line of the output. So if your directory occupies 30MB the last 2 lines of the output would be 30M.

$ du -s
Use this command to displays a summary of the directory size. It is the simplest way to know the total size of the current directory.

$ du -S
This would display the size of the current directory excluding the size of the subdirectories that exist within that directory. So it basically shows you the total size of all the files that exist in the current directory.

$ du --exculde=mp3
Several times it required to exclude some directory in our size calculation. In such cases the above command would display the size of the current directory along with all its subdirectories, but it would exclude all the files having the given pattern present in their filenames.

'df' - finding the disk free space / disk usage

$ df
Hmmm … now “df” command is really useful, and I guess you are going to use it over time. Typing the above command, outputs a table consisting of 6 columns. All the columns are very easy to understand. Remember that the 'Size', 'Used' and 'Avail' columns use kilobytes as the unit. The 'Use%' column shows the usage as a percentage which is also very useful.

$ df -h
Displays the same output as the previous command but the '-h' indicates human readable format. Hence instead of kilobytes as the unit the output would have 'M' for Megabytes and 'G' for Gigabytes.

Example: Linux installed on /dev/hda1
$ df -h | grep /dev/hda1

All right, this is not the only option to check the sizes and free spaces but there are a few more options that can be used with 'du' and 'df' . I will discuss it later.

sort a space-delimited file based on its first column, then the second if the first is the same, and so on:
sort input.txt

sort a huge file (GNU sort ONLY):
sort -S 1500M -t $HOME/tmp input.txt > sorted.txt

sort starting from the third column, skipping the first two columns:
sort +2 input.txt

sort the second column as numbers, descending order; if identical, sort the 3rd as strings, ascending order:
sort -k2,2nr -k3,3 input.txt

sort starting from the 4th character at column 2, as numbers:
sort -k2.4n input.txt

More Linxu sort command information

If you have any sort commands you'd like to share, please add them to our comments section below. For more help, you can also type:

man sort

or

sort --help

on your Unix/Linux system.

The First time when you login to the remotehost from a localhost, it will display the host key not found message and you can give “yes” to continue. The host key of the remote host will be added under .ssh2/hostkeys directory of your home directory, as shown below.

localhost$ ssh -l jit remotehost.example.com

jit@remotehost.example.com password:

remotehost.example.com$

The Second time when you login to the remote host from the localhost, it will prompt only for the password as the remote host key is already added to the known hosts list of the ssh client.

localhost$ ssh -l jit remotehost.example.com
jit@remotehost.example.com password:
remotehost.example.com$

For some reason, if the host key of the remote host is changed after you logged in for the first time, you may get a warning message as shown below. This could be because of various reasons such as 1) Sysadmin upgraded/reinstalled the SSH server on the remote host 2) someone is doing malicious activity etc., The best possible action to take before saying “yes” to the message below, is to call your sysadmin and identify why you got the host key changed message and verify whether it is the correct host key or not.

localhost$ ssh -l jit remotehost.example.com

jit @remotehost.example.com's password:
remotehost$

4. Debug SSH Client:

Sometimes it is necessary to view debug messages to troubleshoot any SSH connection issues. For this purpose, pass -v (lowercase v) option to the ssh as shown below.

Example without debug message:

        localhost$ ssh -l jit remotehost.example.com
        warning: Connecting to remotehost.example.com failed: No address associated to the name
        localhost$

Example with debug message:

        locaclhost$ ssh -v -l jit remotehost.example.com
        debug: SshConfig/sshconfig.c:2838/ssh2_parse_config_ext: Metaconfig parsing stopped at line 3.
        debug: SshConfig/sshconfig.c:637/ssh_config_set_param_verbose: Setting variable 'VerboseMode' to 'FALSE'.
        debug: SshConfig/sshconfig.c:3130/ssh_config_read_file_ext: Read 17 params from config file.
        debug: Ssh2/ssh2.c:1707/main: User config file not found, using defaults. (Looked for '/home/jit/.ssh2/ssh2_config')
        debug: Connecting to remotehost.example.com, port 22... (SOCKS not used)
        warning: Connecting to remotehost.example.com failed: No address associated to

5. Escape Character: (Toggle SSH session, SSH session statistics etc.)

Escape character ~ get’s SSH clients attention and the character following the ~ determines the escape command.
Toggle SSH Session: When you’ve logged on to the remotehost using ssh from the localhost, you may want to come back to the localhost to perform some activity and go back to remote host again. In this case, you don’t need to disconnect the ssh session to the remote host. Instead follow the steps below.

i. Login to remotehost from localhost: localhost$ssh -l jit remotehost
ii. Now you are connected to the remotehost: remotehost$
iii. To come back to the localhost temporarily, type the escape character ~ and Control-Z. When you type ~ you will not see that immediately on the screen until you press and press enter. So, on the remotehost in a new line enter the following key strokes for the below to work: ~

    remotehost$ ~^Z
    [1]+  Stopped                 ssh -l jit remotehost
    localhost$

iv. Now you are back to the localhost and the ssh remotehost client session runs as a typical unix background job, which you can check as shown below:

    localhost$ jobs
    [1]+  Stopped                 ssh -l jit remotehost

v. You can go back to the remote host ssh without entering the password again by bringing the background ssh remotehost session job to foreground on the localhost

    localhost$ fg %1
    ssh -l jit remotehost
    remotehost$

In our day to day research activity, we need to securely copy our data from several to local computer and visa-versa. I am jotting down some of the commonly used SCP command for your future help. Hope you all will like it

What is Secure Copy?

scp allows files to be copied to, from, or between different hosts. It uses ssh for data transfer and provides the same authentication and same level of security as ssh.

Examples

Copy the file "gene.txt" from a remote host to the local host

    $ scp your_username@remotehost.edu:gene.txt /some/local/directory

Copy the file "foobar.txt" from the local host to a remote host

    $ scp gene.txt your_username@remotehost.edu:/some/remote/directory

Copy the directory "chromosome" from the local host to a remote host's directory "bar"

    $ scp -r chromosome your_username@remotehost.edu:/some/remote/directory/bar

Copy the file "gene.txt" from remote host "rh1.edu" to remote host "rh2.edu"

    $ scp your_username@rh1.edu:/some/remote/directory/gene.txt \
    your_username@rh2.edu:/some/remote/directory/

Copying the files "gene.txt" and "cancer.txt" from the local host to your home directory on the remote host

    $ scp gene.txt cancer.txt your_username@remotehost.edu:~

Copy the file "gene.txt" from the local host to a remote host using port 2264

    $ scp -P 2264 gene.txt your_username@remotehost.edu:/some/remote/directory

Copy multiple files from the remote host to your current directory on the local host

    $ scp your_username@remotehost.edu:/some/remote/directory/\{a,b,c\} .

    $ scp your_username@remotehost.edu:~/\{gene.txt,cancer.txt\} .

scp Performance

By default scp uses the Triple-DES cipher to encrypt the data being sent. Using the Blowfish cipher has been shown to increase speed. This can be done by using option -c blowfish in the command line.

    $ scp -c blowfish some_file your_username@remotehost.edu:~

It is often suggested that the -C option for compression should also be used to increase speed. The effect of compression, however, will only significantly increase speed if your connection is very slow. Otherwise it may just be adding extra burden to the CPU. An example of using blowfish and compression:

    $ scp -c blowfish -C local_file your_username@remotehost.edu:~

For knowledge about Linux and their importance amongst bioinformatician plesae read this article "An introduction to Linux for bioinformatics" by Paul Stothard.

Linux cheat sheet at http://bioinformaticsonline.com/file/view/87/linux-cheat-sheet

Please browse for futher useful linux pages on right hand side ...

Basic Galaxy Tutorial

• RNA-Seq tutorial based on Trapnell et al. (2012) Nature Protocols

In this tutorial we cover the concepts of RNA-Seq differential gene expression (DGE) analysis using a very small synthetic dataset from a well studied organism.

Advanced Galaxy Tutorial

• RNA-Seq (Advanced) Tutorial

In this tutorial we compare the performance of three statistically-based differential expression tools:

* CuffDiff

* EdgeR

* DESeq2

Advanced Command Line Tutorial

• Graphical Output with CummeRbund introduces some basic commands using the cummeRbund package of the R programming language

You will need to install R, RStudio and cummeRbund on your PC (explained in the Tutorial). You will learn how to produce graphical output from RNA-Seq analysis previously done using a Cuffdiff analysis.

Variant Detection

Basic Galaxy Tutorial

• Variant Detection tutorial

In this tutorial we cover the concepts of detecting small variants (SNVs and indels) in human genomic DNA using a small set of reads from chromosome 22.

Advanced Galaxy Tutorial

• Variant Detection (Advanced) Tutorial

In this tutorial we compare the performance of three statistically-based variant detection tools:

* SAMtools: Mpileup

* GATK: Unified Genotyper

* FreeBayes

Each of these tools takes as its input a BAM file of aligned reads and generates a list of likely variants in VCF format

Pipelines are for those who are comfortable with using the UNIX command line; and often allow more control over branching and iteration logic.

• WGS/exome GATK-based variant calling pipeline

This is a basic variant-calling and annotation pipeline developed at the Victorian Life Sciences Computation Initiative (VLSCI), University of Melbourne. It is based around BWA, GATK and ENSEMBL and was originally designed for human (or similar) data. The master branch is configured for WGS data; there is an exome branch configured for variant calling in exome data.

To run the pipeline you will need Rubra: https://github.com/bjpop/rubra. Rubra uses the python Ruffus library: http://www.ruffus.org.uk/.

Protocols

• Familial Variant Calling

In this protocol we discuss and outline the process of calling familial related mutations.

• Somatic Variant Calling

In this protocol we discuss and outline the process of identifying somatic variants or mutations.

Assembly

Basic Galaxy Tutorial

• Genome assembly tutorial

In this tutorial we carry out de novo assembly of a microbial genome. We have also written a De novo Genome Assembly for Illumina Data Protocol for a more generic description of the method.

Protocol

• De novo Genome Assembly for Illumina Data

In this protocol we discuss and outline the process of de novo assembly for small to medium sized genomes. Use our Genome assembly tutorial to learn a specific case of using Galaxy to carry out de novo assembly of a microbial genome.

Small RNAs

Basic Galaxy Tutorial

• Quality control for small RNA

This tutorial covers initial steps of the workflow for analysis of short RNA expression such as a quality control of the raw reads, processing of the raw reads for the subsequent analysis and initial quality assessment of the library.

ChIP Seq

Protocol

• ChIP-Seq

In this protocol we discuss ChIP-Seq: a method to analyze the interaction between proteins and DNA.

Amplicons

Protocol

• Amplicon Alignment

In this protocol we discuss and outline the process of aligning custom amplicons using primers for high precision.

Learn Galaxy

Introduction to Galaxy, for those who are very new to Galaxy.

Using Histories and Workflows, for those with some Galaxy knowledge.

The Galaxy project website has many tutorials and screencasts about using Galaxy and the tools, and developing new tools.

Address of the bookmark: https://genome.edu.au/wiki/Learn

To cite BioNumbers please refer to: Milo et al. Nucl. Acids Res. (2010) 38: D750-D753. When using a specific entry from the database it is highly recommended that you also specify the BioNumbers 6 digit ID, e.g. "BNID 100986, Milo et al 2010". 

Address of the bookmark: http://bionumbers.hms.harvard.edu/