'du' – Check the size of a directory

$ du
This command ( du) gives you a list of directories that exist in the current working directory along with their sizes in kilobytes (default). The last line of the output gives you the total size of the current directory including its subdirectories.

$ du /home/jin1
The above command would give you the directory size of the directory /home/david

$ du -h
The same “du”command with some flag gives you a better output than the default one. The option '-h' stands for human readable format. Therefore, in order to print the sizes of the files / directories in your desire notation use this time suffixed with a 'k' if its kilobytes and 'M' if its Megabytes and 'G' if its Gigabytes.

$ du -ah
If you are interested in checking everything present in a folder use above mentioned command. It gives us not only the directories but also all the files that are present in the current directory. The “-a” flag displays the filenames along with the directory names in the output.

$ du -c
This gives you a grand total as the last line of the output. So if your directory occupies 30MB the last 2 lines of the output would be 30M.

$ du -s
Use this command to displays a summary of the directory size. It is the simplest way to know the total size of the current directory.

$ du -S
This would display the size of the current directory excluding the size of the subdirectories that exist within that directory. So it basically shows you the total size of all the files that exist in the current directory.

$ du --exculde=mp3
Several times it required to exclude some directory in our size calculation. In such cases the above command would display the size of the current directory along with all its subdirectories, but it would exclude all the files having the given pattern present in their filenames.

'df' - finding the disk free space / disk usage

$ df
Hmmm … now “df” command is really useful, and I guess you are going to use it over time. Typing the above command, outputs a table consisting of 6 columns. All the columns are very easy to understand. Remember that the 'Size', 'Used' and 'Avail' columns use kilobytes as the unit. The 'Use%' column shows the usage as a percentage which is also very useful.

$ df -h
Displays the same output as the previous command but the '-h' indicates human readable format. Hence instead of kilobytes as the unit the output would have 'M' for Megabytes and 'G' for Gigabytes.

Example: Linux installed on /dev/hda1
$ df -h | grep /dev/hda1

All right, this is not the only option to check the sizes and free spaces but there are a few more options that can be used with 'du' and 'df' . I will discuss it later.

More at https://boards.greenhouse.io/twistbioscience/jobs/3135495?gh_src=9ecc0b941us

For knowledge about Linux and their importance amongst bioinformatician plesae read this article "An introduction to Linux for bioinformatics" by Paul Stothard.

Linux cheat sheet at http://bioinformaticsonline.com/file/view/87/linux-cheat-sheet

Please browse for futher useful linux pages on right hand side ...

Screen is like a window manager for your console. It will allow you to keep multiple terminal sessions running and easily switch between them. It also protects you from disconnection, because the screen session doesn’t end when you get disconnected.

You’ll need to make sure that screen is installed on the server you are connecting to. If that server is Ubuntu or Debian, just use this command:

sudo apt-get install screen

Now you can start a new screen session by just typing screen at the command line. You’ll be shown some information about screen. Hit enter, and you’ll be at a normal prompt.

To disconnect (but leave the session running)

Hit Ctrl + A and then Ctrl + D in immediate succession. You will see the message [detached]

To reconnect to an already running session

screen -r

To reconnect to an existing session, or create a new one if none exists

screen -D -r

To create a new window inside of a running screen session

Hit Ctrl + A and then C in immediate succession. You will see a new prompt.

To switch from one screen window to another

Hit Ctrl + A and then Ctrl + A in immediate succession.

To list open screen windows

Hit Ctrl + A and then W in immediate succession

Tasks:  239 total,   1 running,  238 sleeping,   0 stopped,   0 zombie

This line tells you how many processes are running. If you are using laptops machines it’s not so interesting because you really are the only one using this machine.

Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

This line contains the CPU load. The first two numbers are how busy the system is doing computation (“us” stands for “user”) and how busy the system is doing system-y things like accessing disks or network (“sy” stands for “system”). We’ll talk more about this later.

Mem:   49457320k total,    3492174k used,  14535596k free,    1435148k buffers

This should be easy to understand – how much memory you’re using!

Swap:   539356k total,   28332k used,   836562k free,    29862014k cached

Swap is just on-disk memory that can be used to “swap” out programs from main memory. Again, we’ll talk about this later.:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  1 root      39   19  0  0  0 S  0.0  0.0   246:57.22 kipmi0
  2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0

And... finally! What’s actually running! The two most important numbers are the %CPU and %MEM towards the right, as well as the COMMAND. This tells you how compute- and memory-intensive your program is. Right now, nothing’s running so the numbers aren’t very interesting, but just wait until we run something...

• January 20th, 2016 - New! Bpipe 0.9.9 released!
• Download latest, all
• Documentation
• Mailing List (Google Group)

Bpipe has been published in Bioinformatics! If you use Bpipe, please cite:

Sadedin S, Pope B & Oshlack A, Bpipe: A Tool for Running and Managing Bioinformatics Pipelines, Bioinformatics

Address of the bookmark: http://docs.bpipe.org/

• File System
  ls — list items in current directory
  ls -l — list items in current directory and show in long format to see perimissions, size, and modification date
  ls -a — list all items in current directory, including hidden files
  ls -F — list all items in current directory and show directories with a slash and executables with a star
  ls dir — list all items in directory dir
  cd dir — change directory to dir
  cd .. — go up one directory
  cd / — go to the root directory
  cd ~ — go to to your home directory
  cd - — go to the last directory you were just in
  pwd — show present working directory
  mkdir dir — make directory dir
  rm file — remove file
  rm -r dir — remove directory dir recursively
  cp file1 file2 — copy file1 to file2
  cp -r dir1 dir2 — copy directory dir1 to dir2 recursively
  mv file1 file2 — move (rename) file1 to file2
  ln -s file link — create symbolic link to file
  touch file — create or update file
  cat file — output the contents of file
  less file — view file with page navigation
  head file — output the first 10 lines of file
  tail file — output the last 10 lines of file
  tail -f file — output the contents of file as it grows, starting with the last 10 lines
  vim file — edit file
  alias name 'command' — create an alias for a command
  
• System
  shutdown — shut down machine
  reboot — restart machine
  date — show the current date and time
  whoami — who you are logged in as
  finger user — display information about user
  man command — show the manual for command
  df — show disk usage
  du — show directory space usage
  free — show memory and swap usage
  whereis app — show possible locations of app
  which app — show which app will be run by default
  
• Process Management
  ps — display your currently active processes
  top — display all running processes
  kill pid — kill process id pid
  kill -9 pid — force kill process id pid
  
• Permissions
  ls -l — list items in current directory and show permissions
  chmod ugo file — change permissions of file to ugo - u is the user's permissions, g is the group's permissions, and o is everyone else's permissions. The values of u, g, and o can be any number between 0 and 7.
  7 — full permissions
  6 — read and write only
  5 — read and execute only
  4 — read only
  3 — write and execute only
  2 — write only
  1 — execute only
  0 — no permissions
  chmod 600 file — you can read and write - good for files
  chmod 700 file — you can read, write, and execute - good for scripts
  chmod 644 file — you can read and write, and everyone else can only read - good for web pages
  chmod 755 file — you can read, write, and execute, and everyone else can read and execute - good for programs that you want to share
  
• Networking
  wget file — download a file
  curl file — download a file
  scp user@host:file dir — secure copy a file from remote server to the dir directory on your machine
  scp file user@host:dir — secure copy a file from your machine to the dir directory on a remote server
  scp -r user@host:dir dir — secure copy the directory dir from remote server to the directory dir on your machine
  ssh user@host — connect to host as user
  ssh -p port user@host — connect to host on port as user
  ssh-copy-id user@host — add your key to host for user to enable a keyed or passwordless login
  ping host — ping host and output results
  whois domain — get information for domain
  dig domain — get DNS information for domain
  dig -x host — reverse lookup host
  lsof -i tcp:1337 — list all processes running on port 1337
  
• Searching
  grep pattern files — search for pattern in files
  grep -r pattern dir — search recursively for pattern in dir
  grep -rn pattern dir — search recursively for pattern in dir and show the line number found
  grep -r pattern dir --include='*.ext — search recursively for pattern in dir and only search in files with .ext extension
  command | grep pattern — search for pattern in the output of command
  find file — find all instances of file in real system
  locate file — find all instances of file using indexed database built from the updatedb command. Much faster than find
  sed -i 's/day/night/g' file — find all occurrences of day in a file and replace them with night - s means substitude and g means global - sed also supports regular expressions
  
• Compression
  tar cf file.tar files — create a tar named file.tar containing files
  tar xf file.tar — extract the files from file.tar
  tar czf file.tar.gz files — create a tar with Gzip compression
  tar xzf file.tar.gz — extract a tar using Gzip
  gzip file — compresses file and renames it to file.gz
  gzip -d file.gz — decompresses file.gz back to file
  
• Shortcuts
  ctrl+a — move cursor to beginning of line
  ctrl+f — move cursor to end of line
  alt+f — move cursor forward 1 word
  alt+b — move cursor backward 1 word
  

Read more: https://edu.t-bio.info/amity-university-summer-bioinformatics-program-registrations-are-open/

Part I – Basic Find Commands for Finding Files with Names
1. Find Files Using Name in Current Directory

Find all the files whose name is gene.txt in a current working directory.

# find . -name gene.txt

./gene.txt

2. Find Files Under Home Directory

Find all the files under /home directory with name gene.txt.

# find /home -name gene.txt

/home/gene.txt

3. Find Files Using Name and Ignoring Case

Find all the files whose name is gene.txt and contains both capital and small letters in /home directory.

# find /home -iname gene.txt

./gene.txt
./Gene.txt

4. Find Directories Using Name

Find all directories whose name is Gene in / directory.

# find / -type d -name Gene

/Gene

5. Find fasta Files Using Name

Find all php files whose name is gene.fasta in a current working directory.

# find . -type f -name gene.fasta

./gene.fasta

6. Find all PHP Files in Directory

Find all fasta files in a directory.

# find . -type f -name "*.fasta"

./gene.fasta
./cancer.fasta
./allgene.fasta

Part II – Find Files Based on their Permissions
7. Find Files With 777 Permissions

Find all the files whose permissions are 777.

# find . -type f -perm 0777 -print

8. Find Files Without 777 Permissions

Find all the files without permission 777.

# find / -type f ! -perm 777

9. Find SGID Files with 644 Permissions

Find all the SGID bit files whose permissions set to 644.

# find / -perm 2644

10. Find Sticky Bit Files with 551 Permissions

Find all the Sticky Bit set files whose permission are 551.

# find / -perm 1551

11. Find SUID Files

Find all SUID set files.

# find / -perm /u=s

12. Find SGID Files

Find all SGID set files.

# find / -perm /g+s

13. Find Read Only Files

Find all Read Only files.

# find / -perm /u=r

14. Find Executable Files

Find all Executable files.

# find / -perm /a=x

15. Find Files with 777 Permissions and Chmod to 644

Find all 777 permission files and use chmod command to set permissions to 644.

# find / -type f -perm 0777 -print -exec chmod 644 {} \;

16. Find Directories with 777 Permissions and Chmod to 755

Find all 777 permission directories and use chmod command to set permissions to 755.

# find / -type d -perm 777 -print -exec chmod 755 {} \;

17. Find and remove single File

To find a single file called gene.txt and remove it.

# find . -type f -name "gene.txt" -exec rm -f {} \;

18. Find and remove Multiple File

To find and remove multiple files such as .fa or .gb, then use.

# find . -type f -name "*.fa" -exec rm -f {} \;

OR

# find . -type f -name "*.gb" -exec rm -f {} \;

19. Find all Empty Files

To file all empty files under certain path.

# find /tmp -type f -empty

20. Find all Empty Directories

To file all empty directories under certain path.

# find /tmp -type d -empty

21. File all Hidden Files

To find all hidden files, use below command.

# find /tmp -type f -name ".*"

Part III – Search Files Based On Owners and Groups
22. Find Single File Based on User

To find all or single file called gene.txt under / root directory of owner root.

# find / -user root -name gene.txt

23. Find all Files Based on User

To find all files that belongs to user Rahul under /home directory.

# find /home -user rahul

24. Find all Files Based on Group

To find all files that belongs to group Developer under /home directory.

# find /home -group developer

25. Find Particular Files of User

To find all .txt files of user Rahul under /home directory.

# find /home -user rahul -iname "*.txt"

Part IV – Find Files and Directories Based on Date and Time
26. Find Last 50 Days Modified Files

To find all the files which are modified 50 days back.

# find / -mtime 50

27. Find Last 50 Days Accessed Files

To find all the files which are accessed 50 days back.

# find / -atime 50

28. Find Last 50-100 Days Modified Files

To find all the files which are modified more than 50 days back and less than 100 days.

# find / -mtime +50 –mtime -100

29. Find Changed Files in Last 1 Hour

To find all the files which are changed in last 1 hour.

# find / -cmin -60

30. Find Modified Files in Last 1 Hour

To find all the files which are modified in last 1 hour.

# find / -mmin -60

31. Find Accessed Files in Last 1 Hour

To find all the files which are accessed in last 1 hour.

# find / -amin -60

Part V – Find Files and Directories Based on Size
32. Find 50MB Files

To find all 50MB files, use.

# find / -size 50M

33. Find Size between 50MB – 100MB

To find all the files which are greater than 50MB and less than 100MB.

# find / -size +50M -size -100M

34. Find and Delete 100MB Files

To find all 100MB files and delete them using one single command.

# find / -size +100M -exec rm -rf {} \;

35. Find Specific Files and Delete

Find all .gb files with more than 10MB and delete them using one single command.

# find / -type f -name *.gb -size +10M -exec rm {} \;