BOL: Related items

Monitor running jobs on Linux server

Jitendra Narayan — Fri, 06 Jun 2014 16:18:43 -0500

You as a bioinformatican run lots of program on your servers. Sometime the shared server is also used by your colleague. If server is busy you sometime need to check the running programs and want to monitor the running programs as well. The "top" command will come in handy when you need to find out if things are still running, how long they’ve been running, or how much memory is being used.

‘top’ is very simple to run: type

%% top

You’ll get a screen that looks like this, and is updated regularly:

Simple, right? Heh.

First! Note that you can use ‘q’ or ‘CTRL-C’ to exit from ‘top’.

Now let’s read and understand at each line independently.

The first line:

top - 23:00:48 up 39 days, 2 user, load average: 0.00, 0.00, 0.00

The first line tells you the current time, how long the machine has been up, how many users are logged in, and the short/medium/long-term compute load on the machine. If you run something for a long time, you’ll see these numbers go up. Right now, the machine is basically just sitting there, so these are all close to 0.

The second line:

Tasks: 239 total,   1 running, 238 sleeping,   0 stopped,   0 zombie

This line tells you how many processes are running. If you are using laptops machines it’s not so interesting because you really are the only one using this machine.

Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

This line contains the CPU load. The first two numbers are how busy the system is doing computation (“us” stands for “user”) and how busy the system is doing system-y things like accessing disks or network (“sy” stands for “system”). We’ll talk more about this later.

Mem:   49457320k total,    3492174k used, 14535596k free,    1435148k buffers

This should be easy to understand – how much memory you’re using!

Swap:   539356k total,   28332k used,   836562k free,    29862014k cached

Swap is just on-disk memory that can be used to “swap” out programs from main memory. Again, we’ll talk about this later.:

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
1 root      39 19 0 0 0 S 0.0 0.0   246:57.22 kipmi0
2 root      RT   0     0    0    0 S 0.0 0.0   0:00.00 migration/0

And... finally! What’s actually running! The two most important numbers are the %CPU and %MEM towards the right, as well as the COMMAND. This tells you how compute- and memory-intensive your program is. Right now, nothing’s running so the numbers aren’t very interesting, but just wait until we run something...

SLURM

Jit — Wed, 04 May 2016 05:13:21 -0500

SLURM workload manager software, a free open-source workload manager designed specifically to satisfy the demanding needs of high performance computing.

This page is a HOWTO guide for setting up a SLURM installation, currently focused on a CentOS 7 Linux OS. Please send feedback to Ole.H.Nielsen /at/ fysik.dtu.dk.

See the SLURM homepage (also https://computing.llnl.gov/linux/slurm/).

Address of the bookmark: https://wiki.fysik.dtu.dk/niflheim/SLURM

Binding Site Prediction in Protein !

Poonam Mahapatra — Wed, 25 Apr 2018 04:35:57 -0500

The interaction between proteins and other molecules is fundamental to all biological functions. In this section we include tools that can assist in prediction of interaction sites on protein surface and tools for predicting the structure of the intermolecular complex formed between two or more molecules (docking).

Pockets Identification

CASTp

Automatic Identification of pockets and cavities in proteins structure, and quantitation of their volumes using Delaunay triangulation. Available also as PyMOL plugin

Pocket-Finder

Automatic identification of pockets and cavities in proteins structure, and quantitation of their volumes.

PocketPicker

Grid-based technique for the analysis of protein pockets. PocketPicker available as a plugin for PyMOL

Binding Site Prediction

ConSurf

Identification of functional regions in proteins by surface-mapping of phylogenetic information

CRESCENDO

Identification protein interaction sites. It uses sequence conservation patterns in homologous proteins to distinguish between residues that are conserved due to structural restraints from those due to functional restraints.

Ligand Binding Sites

3DLigandSite

The server utilizes protein-structure prediction to provide structural models of the binding site. Ligands bound to structures are superimposed onto the model and use to predict the binding site.

FINDSITE

A threading-based method for ligand-binding site prediction and functional annotation based on binding-site similarity across superimposed groups of threading templates.

LIGSITE^csc

Prediction of binding site by pocket identification using the Connolly surface and degree of conservation

metaPocketA meta server for ligand-binding site prediction. metaPocket use LIGSITE^csc, PASS, Q-SiteFinder and SURFNET

Installing ELGG on Ubuntu !

Jit — Thu, 25 Nov 2021 01:45:59 -0600

Follow this:

Settings file

Error

Your web server does not have permission to create the settings.php file in your installation directory. You have two choices:

1. Change the permissions on the elgg-config directory of your Elgg installation

2. Copy the file elgg-config/settings.example.php to elgg-config/settings.php and follow the instructions in it for setting your database parameters.

To resolve this --

Change the permision

chmod 777 elgg-config

Web server

Warning

Your server does not support automatic testing of the rewrite rules and your browser does not support checking via JavaScript. You can continue the installation, but you may experience problems with your site. You can manually test the rewrite rules by clicking this link: test. You will see the word success if the rules are working.

create a .httaccess

and provide detail in it

Installation of Elgg

(Note: Elgg requires installing an instance of MySQL.)

Steps to make adjustment and disposal of trial installations easier.

MySQL

At this point, we assume you've set up your MySQL database

Start your mysqld server daemon.
(For the sake of this example, we'll assume we're running on port 9999; you shouldn't actually use this port because only one daemon can run on any port number.)
Login to the server as root user, using the mysql client.
Create a database to store Elgg's tables. We'll call the database elggalpha.
mysql> create database elggalpha;

Grant access to a user — for this example we'll call him/her elggalphauser.

mysql> grant all privileges on elggalpha.* to 'elggalphauser'@'moxie'
    -> identified by 'secretpassword';

Elgg

Type umask 022
Change into your public_html directory (aka folder).
cd public_html
make a directory called elgg
mkdir elgg
Change into the elgg subdirectory of public_html.
cd elgg
download elgg into this directory
you'll end up with a file named
/home/your-user-id/public_html/elgg/elgg-X.X.X.X.zip
(It's not literally X.X.X.X, it might be elgg-1.8.0.1.zip, for example.)
unzip elgg
unzip -q elgg-1.8.0.1.zip
you'll end up with a directory (folder) named
/home/your-user-id/public_html/elgg/elgg-X.X.X.X
make a symbolic link (aka shortcut) to this
ln -s elgg-X.X.X.X alpha
(For example, ln -s elgg-1.8.0.1 alpha)
Give user nobody access to this directory
/usr/misc/bin/acl.sh -r -u nobody elgg-1.8.0.1
create a data directory for elgg (get into your home directory first):
```
cd
mkdir -p elgg/alpha/data
chmod -R 755 elgg/alpha/data
```
Change into the elgg/alpha directory.
cd elgg/alpha
Give user nobody access to this subdirectory data
/usr/misc/bin/acl.sh -u nobody data

In your browser, navigate to your elgg installation
http://moxie.cs.oswego.edu/~your-user-id/elgg/alpha/
You'll have a screen which includes this message:

Welcome

Installing Elgg has 6 simple steps and reading this welcome is the first one!

If you haven't already, read through the installation instructions included with Elgg (or click the instructions link at the bottom of the page).

If you are ready to proceed, click the Next button.

Follow configuration process:

The first screen probably will contain a success message in green and a failure message in pink:
```
Requirements check

Your server failed the requirements check. After you have fixed the below issues, refresh this
page. Check the troubleshooting links at the bottom of this page if you need further assistance.
PHP
```
Your server's PHP satisfies all of Elgg's requirements.
```
Web server
```
We think your server is running the Apache web server. The rewrite test failed and the most likely cause is that AllowOverride is not set to All for Elgg's directory. This prevents Apache from processing the .htaccess file which contains the rewrite rules. A less likely cause is Apache is configured with an alias for your Elgg directory and you need to set the RewriteBase in your .htaccess. There are further instructions in the .htaccess file in your Elgg directory.
```
Database
```
The database requirements are checked when Elgg loads its database.
To correct this, replace the .htaccess file that Elgg created with one that you can edit:
```
cp -ip .htaccess temp-htaccess
rm .htaccess
mv temp-htaccess .htaccess
```
Edit the .htaccess file: Go to approximately line 101 (where it says #RewriteBase /) and add the line:
RewriteBase /~your-user-id/elgg/alpha/
Be sure to save the edited file.
Click the Refresh button.
If this hasn't fixed things, seek professional help.
Click the Next button.

Fill in the parameters appropriate to your MySQL installation.

Database installation

If you haven't already created a database for Elgg, do that now. Then fill in the values below to
initialize the Elgg database.

Database Username

elggalphauser

User that has full priviledges to the MySQL database that you created for Elgg

Database Password

secretpassword

Password for the above database user account

Database Name

elggalpha

Name of the Elgg database

Database Host

^‡

moxie:9999

Hostname of the MySQL server (usually localhost)
‡You might need to use 127.0.0.1:9999 for the host
    if you haven't set up MySQL as we've done.



Database Table Prefix

elgg_

The prefix given to all of Elgg's tables (usually elgg_)

Continue filling in forms as requested.

Configure site

Database has been installed.

We need some information about the site as we configure Elgg. If you haven't created a data directory for Elgg, you need to do so now.

Site Name

My Elgg Site version Alpha

The name of your new Elgg site

Site Email Address

your-user-id@oswego.edu

Email address used by Elgg for communication with users

Site URL

http://moxie.cs.oswego.edu/~your-user-id/elgg/alpha/

The address of the site (Elgg usually guesses this correctly)

Elgg Install Directory

/home/your-user-id/public_html/elgg/elgg.1.8.0.1/

The directory where you put the Elgg code (Elgg usually guesses this correctly)

Data Directory

/home/your-user-id/elgg/alpha/data

The directory that you created for Elgg to save files (the permissions on this directory are checked
when you click Next)

Default Site Access

Public

The default access level for new user created content

Create admin account

Site settings have been saved.

It is now time to create an administrator's account.

Display Name

The name that is displayed on the site for this account

Email Address

Username

Account username used for logging in

Password

Account password must be at least 6 characters long

Password Again

Retype password to confirm

FRODOCK 2.0: fast protein–protein docking server

Neel — Wed, 17 Oct 2018 04:31:30 -0500

frodock: a user-friendly protein–protein docking server based on an improved version of FRODOCK that includes a complementary knowledge-based potential. The web interface provides a very effective tool to explore and select protein–protein models and interactively screen them against experimental distance constraints. The competitive success rates and efficiency achieved allow the retrieval of reliable potential protein–protein binding conformations that can be further refined with more computationally demanding strategies.

Address of the bookmark: http://frodock.chaconlab.org/

Bioinformatics web development course

Jit — Wed, 06 Nov 2019 20:42:48 -0600

This web development course, targeted at Biology and Bioinformatics students, aims at teaching from scratch all the skills needed to setup a fully working Linux web server and to develop and deploy web applications for Bioinformatics.

No previous programming knowledge is assumed. By following this tutorial you will learn the fundamental concepts of programming by using scripting languages: variables, types, arrays, cycles, conditional statements, functions, objects, regular expressions, files reading and manipulation et-cetera.

Address of the bookmark: http://www.cellbiol.com/bioinformatics_web_development/introduction/

HNADOCK: a nucleic acid docking server for modeling RNA/DNA–RNA/DNA 3D complex structures

Poonam Mahapatra — Thu, 04 Jun 2020 23:19:07 -0500

The HNADOCK server is to predict the binding complex structure between two nucleic acid molecules through a hierarchical docking algorihtm of an FFT-based global search strategy and an intrinsic scoring function for nucleic acid interactions. Users are required to provide the three-dimensional (3D) structures of the two molecules to be docked.

Address of the bookmark: http://huanglab.phys.hust.edu.cn/hnadock/

SLURM Commands

Shruti Paniwala — Wed, 06 Jul 2022 07:40:07 -0500

SLURM commands

The following table shows SLURM commands on the SOE cluster.

Command	Description
sbatch	Submit batch scripts to the cluster
scancel	Signal jobs or job steps that are under the control of Slurm.
sinfo	View information about SLURM nodes and partitions.
squeue	View information about jobs located in the SLURM scheduling queue
smap	Graphically view information about SLURM jobs, partitions, and set configurations parameters
sqlog	View information about running and finished jobs
sacct	View resource accounting information for finished and running jobs
sstat	View resource accounting information for running jobs

For more information, run man on the commands above. See some examples below.

1. Info about the partitions and nodes
List all the partitions available to you and the nodes therein:

sinfo

Nodes in state idle can accept new jobs.

Show a partition configuratuin, for example, SOE_main

scontrol show partition=SOE_main

Show current info about a specific node:

scontrol show node=

You can also specify a group of nodes in the command above. For example, if your MPI job is running across soenode05,06,35,36, you can execute the command below to get the info on the nodes you are interested in:

scontrol show node=soenode[05-06,35-36]

An informative parameter in the output to look at would be CPULoad. It allows you to see how your application utilizes the CPUs on the running nodes.

2. Submit scripts
The header in a submit script specifies job name, partition (queue), time limit, memory allocation, number of nodes, number of cores, and files to collect standard output and error at run time, for example

#!/bin/bash

#SBATCH --job-name=OMP_run     # job name, "OMP_run"
#SBATCH --partition=SOE_main   # partition (queue)
#SBATCH -t 0-2:00              # time limit: (D-HH:MM) 
#SBATCH --mem=32000            # memory per node in MB 
#SBATCH --nodes=1              # number of nodes
#SBATCH --ntasks-per-node=16   # number of cores
#SBATCH --output=slurm.out     # file to collect standard output
#SBATCH --error=slurm.err      # file to collect standard errors

If the time limit is not specified in the submit script, SLURM will assign the default run time, 3 days. This means the job will be terminated by SLURM in 72 hrs. The maximum allowed run time is two weeks, 14-0:00.
If the memory limit is not requested, SLURM will assign the default 16 GB. The maximum allowed memory per node is 128 GB. To see how much RAM per node your job is using, you can run commands sacct or sstat to query MaxRSS for the job on the node - see examples below.
Depending on a type of application you need to run, the submit script may contain commands to create a temporary space on a computational node - see the discussion about using the file systems on the cluster.
Then it sets the environment specific to the application and starts the application on one or multiple nodes - see sbatch sample scripts in directory /usr/local/Samples on soemaster1.hpc.rutgers.edu.
You can submit your job to the cluster with sbatch command:

sbatch myscript.sh

3. Query job information
List all currently submitted jobs in running and pending states for a user:

squeue -u

Command squeue can be run with format options to expose specific information, for example, when pending job #706 is scheduled to start running:

squeue -j 706 --format="%S"

START_TIME
2015-04-30T09:54:32

More info can be shown by placing additional format options, for example:

squeue -j 706 --format="%i %P %j %u %T %l %C %S"

JOBID PARTITION   NAME    USER STATE   TIMELIMIT  CPUS START_TIME
706   SOE_main  Par_job_3 mike PENDING 3-00:00:00 64   2015-04-30T09:54:32

To see when all the jobs, pending in the queue, are scheduled to start:

squeue --start

List all running and completed jobs for a user

sqlog -u

sqlog -j

The following appreviations are used for the job states:

       CA   CANCELLED      Job was cancelled.

       CD   COMPLETED      Job completed normally.

       CG   COMPLETING     Job is in the process of completing.

       F    FAILED         Job termined abnormally.

       NF   NODE_FAIL      Job terminated due to node failure.

       PD   PENDING        Job is pending allocation.

       R    RUNNING        Job currently has an allocation.

       S    SUSPENDED      Job is suspended.

       TO   TIMEOUT        Job terminated upon reaching its time limit.

You can specify the fields you would like to see in the output of sqlog:

sqlog --format=list

The command below, for example, provides Job ID, user name, exit state, start date-time, and end date-time for job #2831:

sqlog -j 2831 --format=jid,user,state,start,end

List status info for a currently running job:

sstat -j

A formatted output can be used to gain only a specific info, for example, the maximum resident RAM usage on a node:

sstat --format="JobID,MaxRSS" -j

To get statistics on completed jobs by jobID:

sacct --format="JobID,JobName,MaxRSS,Elapsed" -j

To view the same information for all jobs of a user:

sacct --format="JobID,JobName,MaxRSS,Elapsed" -u

To print a list of fields that can be specified with the --format option:

sacct --helpformat

For example, to get Job ID, Job name, Exit state, start date-time, and end date-time for job #2831:

sacct -j 2831 --format="JobID,JobName,State,Start,End"

Another useful command to gain information about a running job is scontrol:

scontrol show job=

4. Cancel a job
To cancel one job:

scancel

To cancel one job and delete the TMP directory created by the submit script on a node:

sdel

To cancel all the jobs for a user:

scancel -u

To cancel one or more jobs by name:

scancel --name

Look up a biological numbers

Jitendra Narayan — Fri, 23 Aug 2013 03:27:45 -0500

Did you ever need to look up a number like the volume of a cell or the cellular concentration of ATP, only to find yourself spending much more time than you wanted on the Internet or flipping through textbooks - all without much success?

Well, it didn’t happen only to you. It is often surprising how difficult it can be to find concrete biological numbers, even for properties that have been measured numerous times. To help solve this for one and all, BioNumbers (the database of key numbers in molecular biology) was created. Along with the numbers, you'll find the relevant references to the original literature, useful comments, and related numbers.

To cite BioNumbers please refer to: Milo et al. Nucl. Acids Res. (2010) 38: D750-D753. When using a specific entry from the database it is highly recommended that you also specify the BioNumbers 6 digit ID, e.g. "BNID 100986, Milo et al 2010".

Address of the bookmark: http://bionumbers.hms.harvard.edu/

HistoneDB 2.0 – with variants

Anjana — Fri, 03 Jun 2016 05:06:20 -0500

This histone database can be used to explore the diversity of histone proteins and their sequence variants in many organisms. The resource was established to better understand how sequence variation may affect functional and structural features of nucleosomes. To get started, select a histone type to explore its variants.

More at http://www.ncbi.nlm.nih.gov/projects/HistoneDB2.0/index.fcgi/browse/

Address of the bookmark: http://www.ncbi.nlm.nih.gov/projects/HistoneDB2.0/index.fcgi/browse/