When the query file format is fasta, you can specify many threads to process it. It can reduce run time linearly, and use almost equal memory as the original lastz program. This is useful when you lastz a big query file to a huge reference like human whole genome sequence.

The program is an extension on the original lastz program which was written by Bob Harris (the LASTZ guy).

Address of the bookmark: https://github.com/jnarayan81/parallelLastz

Address of the bookmark: https://github.com/AdamCicherski/AlfaPang

Address of the bookmark: http://samtools.sourceforge.net/swlist.shtml

This is a short post giving steps on how to actually install R packages. Let’s suppose you want to install the ggplot2 package. Well nothing could be easier. We just fire up an R shell and type:

> install.packages("ggplot2")

In theory the package should just install, however:

• if you are using Linux and don’t have root access, this command won’t work.
• you will be asked to select your local mirror, i.e. which server should you use to download the package.

Installing packages without root access

First, you need to designate a directory where you will store the downloaded packages. On my machine, I use the directory /data/Rpackages/ After creating a package directory, to install a package we use the command:

> install.packages("ggplot2", lib="/data/Rpackages/")
> library(ggplot2, lib.loc="/data/Rpackages/")


It’s a bit of a pain having to type /data/Rpackages/ all the time. To avoid this burden,  we create a file .Renviron in our home area, and add the line R_LIBS=/data/Rpackages/ to it. This means that whenever you start R, the directory /data/Rpackages/ is added to the list of places to look for R packages and so:

> install.packages("ggplot2")
> library(ggplot2)

just works!

Setting the repository

Every time you install a R package, you are asked which repository R should use. To set the repository and avoid having to specify this at every package install, simply:

• create a file .Rprofile in your home area.
• Add the following piece of code to it:


cat(".Rprofile: Setting UK repositoryn")
r = getOption("repos") # hard code the UK repo for CRAN
r["CRAN"] = "http://cran.uk.r-project.org"
options(repos = r)
rm(r)


I found this tip in a stackoverflow answer .

##################################################
[Mon May 7 11:29:18 2018] Reduction...
#file name genome size contigs heterozygous size [%] heterozygous contigs [%] identity [%] possible joins homozygous size [%] homozygous contigs [%]
/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
test/run1/contigs.fa 163897 245 66377 40.50 221 90.20 94.854 0 97520 59.50 24 9.80

##################################################
[Mon May 7 11:29:29 2018] Estimating parameters of libraries...
Aligning 19504 mates per library...
Insert size statistics Mates orientation stats
FastQ files read length median mean stdev FF FR RF RR
test/5000_1.fq.gz test/5000_2.fq.gz 50 4998 4990.20 721.47 0 4674 0 0
test/600_1.fq.gz test/600_2.fq.gz 100 599 598.63 47.68 0 10000 0 0

##################################################
[Mon May 7 11:29:29 2018] Scaffolding...
iteration 1.1: test/run1/contigs.reduced.fa 24 97520 39.355 17 94157 7321 2195 0 29603
19505 pairs. 17302 passed filtering [88.71%]. 1627 in different contigs [8.34%].
1526 pairs. 558 in different contigs [36.57%].
iteration 1.2: test/run1/_sspace.1.1.fa 3 97626 39.344 3 97626 87536 6063 821 87536
19505 pairs. 17607 passed filtering [90.27%]. 182 in different contigs [0.93%].
1077 pairs. 124 in different contigs [11.51%].
iteration 2.1: test/run1/_sspace.1.2.fa 3 97626 39.344 3 97626 87536 6063 821 87536
19505 pairs. 15112 passed filtering [77.48%]. 1295 in different contigs [6.64%].
3417 pairs. 396 in different contigs [11.59%].
iteration 2.2: test/run1/_sspace.2.1.fa 1 99133 39.344 1 99133 99133 99133 2328 99133
19505 pairs. 15152 passed filtering [77.68%]. 0 in different contigs [0.00%].
3398 pairs. 0 in different contigs [0.00%].

##################################################
[Mon May 7 11:29:34 2018] Gap closing...
iteration 1.1: test/run1/scaffolds.fa 1 99133 39.344 1 99133 99133 99133 2328 99133

##################################################
[Mon May 7 11:29:35 2018] Final reduction...
#file name genome size contigs heterozygous size [%] heterozygous contigs [%] identity [%] possible joins homozygous size [%] homozygous contigs [%]
[WARNING] Nothing reduced!
test/run1/scaffolds.filled.fa 99390 1 0 0.00 0 0.00 0.000 0 99390 100.00 1 100.00

##################################################
[Mon May 7 11:29:35 2018] Reporting statistics...
#fname contigs bases GC [%] contigs >1kb bases in contigs >1kb N50 N90 Ns longest
test/contigs.fa 245 163897 40.298 24 117391 3975 233 0 29603
test/run1/contigs.fa 245 163897 40.298 24 117391 3975 233 0 29603
test/run1/contigs.reduced.fa 24 97520 39.355 17 94157 7321 2195 0 29603
test/run1/_sspace.1.1.fa 3 97626 39.344 3 97626 87536 6063 821 87536
test/run1/_sspace.1.2.fa 3 97626 39.344 3 97626 87536 6063 821 87536
test/run1/_sspace.2.1.fa 1 99133 39.344 1 99133 99133 99133 2328 99133
test/run1/_sspace.2.2.fa 1 99133 39.344 1 99133 99133 99133 2328 99133
test/run1/scaffolds.fa 1 99133 39.344 1 99133 99133 99133 2328 99133
test/run1/_gapcloser.1.1.fa 1 99390 39.689 1 99390 99390 99390 2 99390
test/run1/scaffolds.filled.fa 1 99390 39.689 1 99390 99390 99390 2 99390
test/run1/scaffolds.reduced.fa 1 99390 39.689 1 99390 99390 99390 2 99390

##################################################
[Mon May 7 11:29:35 2018] Cleaning-up...
#Time elapsed: 0:00:17.376924

The First time when you login to the remotehost from a localhost, it will display the host key not found message and you can give “yes” to continue. The host key of the remote host will be added under .ssh2/hostkeys directory of your home directory, as shown below.

localhost$ ssh -l jit remotehost.example.com

jit@remotehost.example.com password:

remotehost.example.com$

The Second time when you login to the remote host from the localhost, it will prompt only for the password as the remote host key is already added to the known hosts list of the ssh client.

localhost$ ssh -l jit remotehost.example.com
jit@remotehost.example.com password:
remotehost.example.com$

For some reason, if the host key of the remote host is changed after you logged in for the first time, you may get a warning message as shown below. This could be because of various reasons such as 1) Sysadmin upgraded/reinstalled the SSH server on the remote host 2) someone is doing malicious activity etc., The best possible action to take before saying “yes” to the message below, is to call your sysadmin and identify why you got the host key changed message and verify whether it is the correct host key or not.

localhost$ ssh -l jit remotehost.example.com

jit @remotehost.example.com's password:
remotehost$

4. Debug SSH Client:

Sometimes it is necessary to view debug messages to troubleshoot any SSH connection issues. For this purpose, pass -v (lowercase v) option to the ssh as shown below.

Example without debug message:

        localhost$ ssh -l jit remotehost.example.com
        warning: Connecting to remotehost.example.com failed: No address associated to the name
        localhost$

Example with debug message:

        locaclhost$ ssh -v -l jit remotehost.example.com
        debug: SshConfig/sshconfig.c:2838/ssh2_parse_config_ext: Metaconfig parsing stopped at line 3.
        debug: SshConfig/sshconfig.c:637/ssh_config_set_param_verbose: Setting variable 'VerboseMode' to 'FALSE'.
        debug: SshConfig/sshconfig.c:3130/ssh_config_read_file_ext: Read 17 params from config file.
        debug: Ssh2/ssh2.c:1707/main: User config file not found, using defaults. (Looked for '/home/jit/.ssh2/ssh2_config')
        debug: Connecting to remotehost.example.com, port 22... (SOCKS not used)
        warning: Connecting to remotehost.example.com failed: No address associated to

5. Escape Character: (Toggle SSH session, SSH session statistics etc.)

Escape character ~ get’s SSH clients attention and the character following the ~ determines the escape command.
Toggle SSH Session: When you’ve logged on to the remotehost using ssh from the localhost, you may want to come back to the localhost to perform some activity and go back to remote host again. In this case, you don’t need to disconnect the ssh session to the remote host. Instead follow the steps below.

i. Login to remotehost from localhost: localhost$ssh -l jit remotehost
ii. Now you are connected to the remotehost: remotehost$
iii. To come back to the localhost temporarily, type the escape character ~ and Control-Z. When you type ~ you will not see that immediately on the screen until you press and press enter. So, on the remotehost in a new line enter the following key strokes for the below to work: ~

    remotehost$ ~^Z
    [1]+  Stopped                 ssh -l jit remotehost
    localhost$

iv. Now you are back to the localhost and the ssh remotehost client session runs as a typical unix background job, which you can check as shown below:

    localhost$ jobs
    [1]+  Stopped                 ssh -l jit remotehost

v. You can go back to the remote host ssh without entering the password again by bringing the background ssh remotehost session job to foreground on the localhost

    localhost$ fg %1
    ssh -l jit remotehost
    remotehost$

Part I – Basic Find Commands for Finding Files with Names
1. Find Files Using Name in Current Directory

Find all the files whose name is gene.txt in a current working directory.

# find . -name gene.txt

./gene.txt

2. Find Files Under Home Directory

Find all the files under /home directory with name gene.txt.

# find /home -name gene.txt

/home/gene.txt

3. Find Files Using Name and Ignoring Case

Find all the files whose name is gene.txt and contains both capital and small letters in /home directory.

# find /home -iname gene.txt

./gene.txt
./Gene.txt

4. Find Directories Using Name

Find all directories whose name is Gene in / directory.

# find / -type d -name Gene

/Gene

5. Find fasta Files Using Name

Find all php files whose name is gene.fasta in a current working directory.

# find . -type f -name gene.fasta

./gene.fasta

6. Find all PHP Files in Directory

Find all fasta files in a directory.

# find . -type f -name "*.fasta"

./gene.fasta
./cancer.fasta
./allgene.fasta

Part II – Find Files Based on their Permissions
7. Find Files With 777 Permissions

Find all the files whose permissions are 777.

# find . -type f -perm 0777 -print

8. Find Files Without 777 Permissions

Find all the files without permission 777.

# find / -type f ! -perm 777

9. Find SGID Files with 644 Permissions

Find all the SGID bit files whose permissions set to 644.

# find / -perm 2644

10. Find Sticky Bit Files with 551 Permissions

Find all the Sticky Bit set files whose permission are 551.

# find / -perm 1551

11. Find SUID Files

Find all SUID set files.

# find / -perm /u=s

12. Find SGID Files

Find all SGID set files.

# find / -perm /g+s

13. Find Read Only Files

Find all Read Only files.

# find / -perm /u=r

14. Find Executable Files

Find all Executable files.

# find / -perm /a=x

15. Find Files with 777 Permissions and Chmod to 644

Find all 777 permission files and use chmod command to set permissions to 644.

# find / -type f -perm 0777 -print -exec chmod 644 {} \;

16. Find Directories with 777 Permissions and Chmod to 755

Find all 777 permission directories and use chmod command to set permissions to 755.

# find / -type d -perm 777 -print -exec chmod 755 {} \;

17. Find and remove single File

To find a single file called gene.txt and remove it.

# find . -type f -name "gene.txt" -exec rm -f {} \;

18. Find and remove Multiple File

To find and remove multiple files such as .fa or .gb, then use.

# find . -type f -name "*.fa" -exec rm -f {} \;

OR

# find . -type f -name "*.gb" -exec rm -f {} \;

19. Find all Empty Files

To file all empty files under certain path.

# find /tmp -type f -empty

20. Find all Empty Directories

To file all empty directories under certain path.

# find /tmp -type d -empty

21. File all Hidden Files

To find all hidden files, use below command.

# find /tmp -type f -name ".*"

Part III – Search Files Based On Owners and Groups
22. Find Single File Based on User

To find all or single file called gene.txt under / root directory of owner root.

# find / -user root -name gene.txt

23. Find all Files Based on User

To find all files that belongs to user Rahul under /home directory.

# find /home -user rahul

24. Find all Files Based on Group

To find all files that belongs to group Developer under /home directory.

# find /home -group developer

25. Find Particular Files of User

To find all .txt files of user Rahul under /home directory.

# find /home -user rahul -iname "*.txt"

Part IV – Find Files and Directories Based on Date and Time
26. Find Last 50 Days Modified Files

To find all the files which are modified 50 days back.

# find / -mtime 50

27. Find Last 50 Days Accessed Files

To find all the files which are accessed 50 days back.

# find / -atime 50

28. Find Last 50-100 Days Modified Files

To find all the files which are modified more than 50 days back and less than 100 days.

# find / -mtime +50 –mtime -100

29. Find Changed Files in Last 1 Hour

To find all the files which are changed in last 1 hour.

# find / -cmin -60

30. Find Modified Files in Last 1 Hour

To find all the files which are modified in last 1 hour.

# find / -mmin -60

31. Find Accessed Files in Last 1 Hour

To find all the files which are accessed in last 1 hour.

# find / -amin -60

Part V – Find Files and Directories Based on Size
32. Find 50MB Files

To find all 50MB files, use.

# find / -size 50M

33. Find Size between 50MB – 100MB

To find all the files which are greater than 50MB and less than 100MB.

# find / -size +50M -size -100M

34. Find and Delete 100MB Files

To find all 100MB files and delete them using one single command.

# find / -size +100M -exec rm -rf {} \;

35. Find Specific Files and Delete

Find all .gb files with more than 10MB and delete them using one single command.

# find / -type f -name *.gb -size +10M -exec rm {} \;

Shell history search command

Type history at a shell prompt:
$ history

It will display the list of all used commandline history with an serial number.

To search particular command, enter:
$ history | grep command-name
$ history | egrep -i 'scp|ssh|ftp'
Emacs Line-Edit Mode Command History Searching

To get previous command containing string, hit [CTRL]+[r] followed by search string:

(reverse-i-search):

To get previous command, hit [CTRL]+[p]. You can also use up arrow key.

CTRL-p

To get next command, hit [CTRL]+[n]. You can also use down arrow key.

CTRL-n

fc command

Apart from hostory command there are fc command to extract the command from history. The fc stands for either "find command" or "fix command.

For example list last 10 command, enter:
$ fc -l 10
To list commands 130 through 150, enter:
$ fc -l 130 150
To list all commands since the last command beginning with ssh, enter:
$ fc -l ssh
You can edit commands 1 through 5 using vi text editor, enter:
$ fc -e vi 1 5

Delete command history

The -c option causes the history list to be cleared by deleting all of the entries:
$ history -c

• File System
  ls — list items in current directory
  ls -l — list items in current directory and show in long format to see perimissions, size, and modification date
  ls -a — list all items in current directory, including hidden files
  ls -F — list all items in current directory and show directories with a slash and executables with a star
  ls dir — list all items in directory dir
  cd dir — change directory to dir
  cd .. — go up one directory
  cd / — go to the root directory
  cd ~ — go to to your home directory
  cd - — go to the last directory you were just in
  pwd — show present working directory
  mkdir dir — make directory dir
  rm file — remove file
  rm -r dir — remove directory dir recursively
  cp file1 file2 — copy file1 to file2
  cp -r dir1 dir2 — copy directory dir1 to dir2 recursively
  mv file1 file2 — move (rename) file1 to file2
  ln -s file link — create symbolic link to file
  touch file — create or update file
  cat file — output the contents of file
  less file — view file with page navigation
  head file — output the first 10 lines of file
  tail file — output the last 10 lines of file
  tail -f file — output the contents of file as it grows, starting with the last 10 lines
  vim file — edit file
  alias name 'command' — create an alias for a command
  
• System
  shutdown — shut down machine
  reboot — restart machine
  date — show the current date and time
  whoami — who you are logged in as
  finger user — display information about user
  man command — show the manual for command
  df — show disk usage
  du — show directory space usage
  free — show memory and swap usage
  whereis app — show possible locations of app
  which app — show which app will be run by default
  
• Process Management
  ps — display your currently active processes
  top — display all running processes
  kill pid — kill process id pid
  kill -9 pid — force kill process id pid
  
• Permissions
  ls -l — list items in current directory and show permissions
  chmod ugo file — change permissions of file to ugo - u is the user's permissions, g is the group's permissions, and o is everyone else's permissions. The values of u, g, and o can be any number between 0 and 7.
  7 — full permissions
  6 — read and write only
  5 — read and execute only
  4 — read only
  3 — write and execute only
  2 — write only
  1 — execute only
  0 — no permissions
  chmod 600 file — you can read and write - good for files
  chmod 700 file — you can read, write, and execute - good for scripts
  chmod 644 file — you can read and write, and everyone else can only read - good for web pages
  chmod 755 file — you can read, write, and execute, and everyone else can read and execute - good for programs that you want to share
  
• Networking
  wget file — download a file
  curl file — download a file
  scp user@host:file dir — secure copy a file from remote server to the dir directory on your machine
  scp file user@host:dir — secure copy a file from your machine to the dir directory on a remote server
  scp -r user@host:dir dir — secure copy the directory dir from remote server to the directory dir on your machine
  ssh user@host — connect to host as user
  ssh -p port user@host — connect to host on port as user
  ssh-copy-id user@host — add your key to host for user to enable a keyed or passwordless login
  ping host — ping host and output results
  whois domain — get information for domain
  dig domain — get DNS information for domain
  dig -x host — reverse lookup host
  lsof -i tcp:1337 — list all processes running on port 1337
  
• Searching
  grep pattern files — search for pattern in files
  grep -r pattern dir — search recursively for pattern in dir
  grep -rn pattern dir — search recursively for pattern in dir and show the line number found
  grep -r pattern dir --include='*.ext — search recursively for pattern in dir and only search in files with .ext extension
  command | grep pattern — search for pattern in the output of command
  find file — find all instances of file in real system
  locate file — find all instances of file using indexed database built from the updatedb command. Much faster than find
  sed -i 's/day/night/g' file — find all occurrences of day in a file and replace them with night - s means substitude and g means global - sed also supports regular expressions
  
• Compression
  tar cf file.tar files — create a tar named file.tar containing files
  tar xf file.tar — extract the files from file.tar
  tar czf file.tar.gz files — create a tar with Gzip compression
  tar xzf file.tar.gz — extract a tar using Gzip
  gzip file — compresses file and renames it to file.gz
  gzip -d file.gz — decompresses file.gz back to file
  
• Shortcuts
  ctrl+a — move cursor to beginning of line
  ctrl+f — move cursor to end of line
  alt+f — move cursor forward 1 word
  alt+b — move cursor backward 1 word