The following table shows SLURM commands on the SOE cluster.

For more information, run man on the commands above. See some examples below.

1. Info about the partitions and nodes
List all the partitions available to you and the nodes therein:

sinfo

Nodes in state idle can accept new jobs.

Show a partition configuratuin, for example, SOE_main

scontrol show partition=SOE_main

Show current info about a specific node:

scontrol show node=<nodename>

You can also specify a group of nodes in the command above. For example, if your MPI job is running across soenode05,06,35,36, you can execute the command below to get the info on the nodes you are interested in:

scontrol show node=soenode[05-06,35-36]

An informative parameter in the output to look at would be CPULoad. It allows you to see how your application utilizes the CPUs on the running nodes.

2. Submit scripts
The header in a submit script specifies job name, partition (queue), time limit, memory allocation, number of nodes, number of cores, and files to collect standard output and error at run time, for example

#!/bin/bash

#SBATCH --job-name=OMP_run     # job name, "OMP_run"
#SBATCH --partition=SOE_main   # partition (queue)
#SBATCH -t 0-2:00              # time limit: (D-HH:MM) 
#SBATCH --mem=32000            # memory per node in MB 
#SBATCH --nodes=1              # number of nodes
#SBATCH --ntasks-per-node=16   # number of cores
#SBATCH --output=slurm.out     # file to collect standard output
#SBATCH --error=slurm.err      # file to collect standard errors

If the time limit is not specified in the submit script, SLURM will assign the default run time, 3 days. This means the job will be terminated by SLURM in 72 hrs. The maximum allowed run time is two weeks, 14-0:00.
If the memory limit is not requested, SLURM will assign the default 16 GB. The maximum allowed memory per node is 128 GB. To see how much RAM per node your job is using, you can run commands sacct or sstat to query MaxRSS for the job on the node - see examples below.
Depending on a type of application you need to run, the submit script may contain commands to create a temporary space on a computational node - see the discussion about using the file systems on the cluster.
Then it sets the environment specific to the application and starts the application on one or multiple nodes - see sbatch sample scripts in directory /usr/local/Samples on soemaster1.hpc.rutgers.edu.
You can submit your job to the cluster with sbatch command:

sbatch myscript.sh

3. Query job information
List all currently submitted jobs in running and pending states for a user:

squeue -u <username>

Command squeue can be run with format options to expose specific information, for example, when pending job #706 is scheduled to start running:

squeue -j 706 --format="%S"

START_TIME
2015-04-30T09:54:32

More info can be shown by placing additional format options, for example:

squeue -j 706 --format="%i %P %j %u %T %l %C %S"

JOBID PARTITION   NAME    USER STATE   TIMELIMIT  CPUS START_TIME
706   SOE_main  Par_job_3 mike PENDING 3-00:00:00 64   2015-04-30T09:54:32

To see when all the jobs, pending in the queue, are scheduled to start:

squeue --start 

List all running and completed jobs for a user

sqlog -u <username>

or

sqlog -j <JobID>

The following appreviations are used for the job states:

       CA   CANCELLED      Job was cancelled.

       CD   COMPLETED      Job completed normally.

       CG   COMPLETING     Job is in the process of completing.

       F    FAILED         Job termined abnormally.

       NF   NODE_FAIL      Job terminated due to node failure.

       PD   PENDING        Job is pending allocation.

       R    RUNNING        Job currently has an allocation.

       S    SUSPENDED      Job is suspended.

       TO   TIMEOUT        Job terminated upon reaching its time limit.

You can specify the fields you would like to see in the output of sqlog:

sqlog --format=list

The command below, for example, provides Job ID, user name, exit state, start date-time, and end date-time for job #2831:

sqlog -j 2831 --format=jid,user,state,start,end

List status info for a currently running job:

sstat -j <jobid>

A formatted output can be used to gain only a specific info, for example, the maximum resident RAM usage on a node:

sstat --format="JobID,MaxRSS" -j <jobid>

To get statistics on completed jobs by jobID:

sacct --format="JobID,JobName,MaxRSS,Elapsed" -j <jobid>

To view the same information for all jobs of a user:

sacct --format="JobID,JobName,MaxRSS,Elapsed" -u <username>

To print a list of fields that can be specified with the --format option:

sacct --helpformat

For example, to get Job ID, Job name, Exit state, start date-time, and end date-time for job #2831:

sacct -j 2831 --format="JobID,JobName,State,Start,End"

Another useful command to gain information about a running job is scontrol:

scontrol show job=<jobid>

4. Cancel a job
To cancel one job:

scancel <jobid>

To cancel one job and delete the TMP directory created by the submit script on a node:

sdel <jobid>

To cancel all the jobs for a user:

scancel -u <username>

To cancel one or more jobs by name:

scancel --name <myJobName>

Compare structural and functional annotations of proteins between sequenced genomes.

In this tutorial, we will show you how to install and configure Elgg social networking platform on Ubuntu 20.04.

Prerequisites

• A fresh Ubuntu 20.04 VPS on the Atlantic.net Cloud Platform
• A valid domain name pointed to your server IP
• A root password configured on your server

Step 1 – Create Atlantic.Net Cloud Server

First, log in to your Atlantic.Net Cloud Server. Create a new server, choosing Ubuntu 20.04 as the operating system with at least 2GB RAM. Connect to your Cloud Server via SSH and log in using the credentials highlighted at the top of the page.

Once you are logged in to your Ubuntu 20.04 server, run the following command to update your base system with the latest available packages.

apt-get update -y

Step 2 – Install Apache, MariaDB and PHP

Elgg runs on Apache web server, is written in PHP, and uses MySQL/MariaDB as a database backend, so you will need to install the Apache, MariaDB, PHP and other required PHP extensions to your server. You can install all of them with the following command:

apt-get install apache2 mariadb-server php libapache2-mod-php php-common php-sqlite3 php-curl 
php-intl php-mbstring php-xmlrpc php-mysql php-gd php-xml php-cli php-zip unzip wget -y

After installing all the packages, edit the php.ini file and change some recommended settings.

nano /etc/php/7.4/apache2/php.ini

Change the following values:

max_execution_time = 300
memory_limit = 512M
upload_max_filesize = 100M
date.timezone = Asia/Kolkata

Save and close the file, then restart the Apache service to apply the configuration changes.

systemctl restart apache2

Step 3 – Create a Database for Elgg

Next, you will need to create a database and user for Elgg. First, log in to MySQL shell with the following command:

mysql

Once logged in, create a database and user with the following command:

CREATE DATABASE elgg;
CREATE USER 'elgg'@'localhost' IDENTIFIED BY 'secure-password';

Next, grant all the privileges to the elgg database with the following command:

GRANT ALL ON elgg.* TO 'elgg'@'localhost' IDENTIFIED BY 'secure-password' WITH GRANT 
OPTION;

Next, flush the privileges and exit from the MariaDB shell with the following command:

FLUSH PRIVILEGES;
EXIT;

At this point, the MariaDB database is created for Elgg.

Step 4 – Install Elgg

First, download the latest version of Elgg from its official website using the following command:

wget https://elgg.org/download/elgg-3.3.13.zip

Once the download is completed, unzip the downloaded file with the following command:

unzip elgg-3.3.13.zip

Next, move the extracted directory to the Apache root directory:

mv elgg-3.3.13 /var/www/html/elgg

Next, create a data directory and set proper ownership and permissions to the Elgg directory:

mkdir /var/www/html/data
chown -R www-data:www-data /var/www/html/elgg
chown -R www-data:www-data /var/www/html/data
chmod -R 755 /var/www/html/elgg

Once you are finished, you can proceed to the next step.

Step 5 – Configure Apache for Elgg

Next, you will need to configure Apache to serve Elgg. You can configure it by creating a new Apache virtual host configuration file:

nano /etc/apache2/sites-available/elgg.conf

Add the following lines:

<VirtualHost *:80>
ServerAdmin admin@example.com
DocumentRoot /var/www/html/elgg/
ServerName elgg.example.com
Options FollowSymLinks
AllowOverride All
ErrorLog /var/log/apache2/elgg-error_log
CustomLog /var/log/apache2/elgg-access_log common
</VirtualHost>

Save and close the file, then enable the virtual host and Apache rewrite module with the following command:

a2ensite elgg.conf
a2enmod rewrite

Finally, restart the Apache service to apply the changes:

systemctl restart apache2

Step 6 – Access Elgg Web Interface

Now, open your web browser and access the Elgg web interface using the URL http://elgg.example.com. You should see the Elgg welcome screen:

Classificier of reads from metagenomic sequencing experiments.

•  Kawulok, J., Deorowicz, S., CoMeta: Classification of Metagenomes Using k-mers, PLOS ONE, 2015; 10(4):1–23,

CoMSA

Compressor of multiple sequence alignments of proteins.

•  Deorowicz, S., Walczyszyn, J., Debudaj-Grabysz, A., CoMSA: compression of protein multiple sequence alignment files, Bioinformatics, 2019; 35(2):22–234,

DSRC

Compressor of sequencing reads.

•  Roguski, L., Deorowicz, S., DSRC 2: Industry-oriented compression of FASTQ files, Bioinformatics, 2014; 30(15):2213–2215,
•  Deorowicz, S., Grabowski, Sz., Compression of DNA sequences in FASTQ format, Bioinformatics, 2011; 27(6):860–862,

FAMSA

Multiple sequence alignment designed for huge families of proteins (even containing hundreds of thousands of sequences).

•  Deorowicz, S., Debudaj-Grabysz, A., Gudys, A., FAMSA: Fast and accurate multiple sequence alignment of huge protein families, Scientific Reports, 2016; 6(33964):

FaStore

Compressor of FASTQ files.

•  Roguski, L., Ochoa, I., Hernaez, M., Deorowicz, S., FaStore - a space-saving solution for raw sequencing data, Bioinformatics, 2018; 34(16):2748–2756,

FQSqueezer

Experimental high-end compressor of FASTQ files.

•  Deorowicz, S., FQSqueezer: k-mer-based compression of sequencing data, Scientific Reports, 2020; 10(578):

GDC

Compressor of collections of genome sequences.

•  Deorowicz, S., Danek, A., Niemiec, M., GDC 2: Compression of large collections of genomes, Scientific Reports, 2015; 5(11565):1–12,
•  Deorowicz, S., Grabowski, Sz., Robust relative compression of genomes with random access, Bioinformatics, 2011; 27(21):2979–2986,

GTC

Genotype databases compressor with support for fast queries.

•  Danek, A., Deorowicz, S., GTC: how to maintain huge genotype collections in a compressed form, Bioinformatics, 2018; 34(11):1834–1840,

GTShark

Genotypes compressor.

•  Deorowicz, S., Danek, A., GTShark: Genotype compression in large projects, Bioinformatics, 2019; 35(22):4791–4793,

KMC

Memory frugal k-mer counter.

•  Kokot, M., Długosz, M., Deorowicz, S., KMC 3: counting and manipulating k -mer statistics, Bioinformatics, 2017; 33(17):2759–2761,
•  Deorowicz, S., Kokot, M., Grabowski, Sz., Debudaj-Grabysz, A., KMC 2: Fast and resource-frugal k-mer counting, Bioinformatics, 2015; 31(10):1569–1576,
•  Deorowicz, S., Debudaj-Grabysz, A., Grabowski, Sz., Disk-based k-mer counting on a PC, BMC Bioinformatics, 2013; 14():Article no. 160,

Kmer-db

Tool for estimation of evolutionary distances in a collection of genomes.

•  Deorowicz, S., Gudys, A., Dlugosz, M., Kokot, M., Danek, A., Kmer-db: instant evolutionary distance estimation, Bioinformatics, 2019; 35(1):133–136,

MuGI

Index allowing queries for a collection of multiple genome sequences.

•  Danek, A., Deorowicz, S., Grabowski, Sz., Indexes of Large Genome Collections on a PC, PLOS ONE, 2014; 9(10):e109384,

ORCOM

Experimental compressor of sequencing reads.

•  Grabowski, Sz., Deorowicz, S., Roguski, L., Disk-based compression of data from genome sequencing, Bioinformatics, 2014; 31(9):1389–1395,

PgSA

Index allowing queries for a collection of sequencing reads.

•  Kowalski, T., Grabowski, Sz., Deorowicz, S., Indexing arbitrary-length k-mers in sequencing reads, PLOS ONE, 2015; 10(7):1–16,

QuickProbs

Multiple sequence alignment designed especially for GPU.

•  Gudys, A., Deorowicz, S., QuickProbs 2: towards rapid construction of high-quality alignments of large protein families, Scientific Reports, 2017; 7(41553):
•  Gudys, A., Deorowicz, S., QuickProbs – A Fast Multiple Sequence Alignment Algorithm Designed for Graphics Processors, PLOS ONE, 2014; 9(2):e88901,

RECKONER

Read error corrector.

•  Maciej Długosz, M., Deorowicz, S., RECKONER: read error corrector based on KMC, Bioinformatics, 2017; 33(7):1086–1089,

TGC

Compressor of collections of genomes given in Variant Call Format (VCF) files.

•  Deorowicz, S., Danek, A., Grabowski, Sz., Genome compression: a novel approach for large collections, Bioinformatics, 2013; 29(20):2572–2578,

VCFShark

Compressor of VCF files.

•  Deorowicz, S., Danek, A., GTShark: Genotype compression in large projects, biorxiv.org, 2020; ():

Whisper

Experimental mapper of whole genome sequencing data.

•  Deorowicz, S., Gudys, A., Whisper 2: indel-sensitive short read mapping, bioRxiv.org, 2019; :
•  Deorowicz, S., Debudaj-Grabysz, A., Gudys, A., Grabowski, Sz., Whisper: read sorting allows robust robust mapping of DNA sequencing data, Bioinformatics, 2019; 35(12):2043–2050,
•  Deorowicz, S., Debudaj-Grabysz, A., Gudys, A., Grabowski, Sz., Robust mapping of whole genome sequencing data, Poster at The Biology of Genomes Conference, 2017;

1. Merge re-sequenced FastQ files (cat)
2. Read QC (FastQC)
3. Adapter trimming (fastp)
4. Removal of host reads (Kraken 2; optional)
5. Variant calling
   1. Read alignment (Bowtie 2)
   2. Sort and index alignments (SAMtools)
   3. Primer sequence removal (iVar; amplicon data only)
   4. Duplicate read marking (picard; optional)
   5. Alignment-level QC (picard, SAMtools)
   6. Genome-wide and amplicon coverage QC plots (mosdepth)
   7. Choice of multiple variant calling and consensus sequence generation routes (iVar variants and consensus; default for amplicon data || BCFTools, BEDTools; default for metagenomics data)
      • Variant annotation (SnpEff, SnpSift)
      • Consensus assessment report (QUAST)
      • Lineage analysis (Pangolin)
      • Clade assignment, mutation calling and sequence quality checks (Nextclade)
      • Individual variant screenshots with annotation tracks (ASCIIGenome)
   8. Intersect variants across callers (BCFTools)
6. De novo assembly
   1. Primer trimming (Cutadapt; amplicon data only)
   2. Choice of multiple assembly tools (SPAdes || Unicycler || minia)
      • Blast to reference genome (blastn)
      • Contiguate assembly (ABACAS)
      • Assembly report (PlasmidID)
      • Assembly assessment report (QUAST)
7. Present QC and visualisation for raw read, alignment, assembly and variant calling results (MultiQC)

The variant should be transmitted from its original location to another "geographically distinct population"—say, another country or a province of a large and populous country.
It should differ from its ancestor by at least one nucleotide.
At least 95% of its genetic code should have been sequenced at least five times from different samples.

• data = mock data for the snakefile to use
• Snakefile = name of the snakemake “formula” file
  • Note: The default file that snakemake looks for in the current working directory is the Snakefile. If you would like to override that you can specify it following the -s
    • snakemake -s snakefile.py
• envs = directory for storing the conda environments that the workflow will use.
• scripts = directory for storing python scripts called by the snakemake formula.
• config.json = json format file with extra parameters for our snakemake file to use.
• cluster.json = json format file with specification for running on the HPC
• samples.txt = file we will use later relating to the config.json file.

Run the snakemake file as a dry run (the example workflow shown above).

• This will build a DAG of the jobs to be run without actually executing them.
• snakemake --dry-run

User can execute rules of interest.

• snakemake --dry-run all VS. snakemake --dry-run call VS. snakemake --dry-run bwa

Run the snakemake file in order to produce an image of the DAG of jobs to be run.

• snakemake --dag | dot -Tsvg > dag.svg OR snakemake --dag | dot -Tsvg > dag.svg

Run the snakemake (this time not as a dry run)

1. snakemake --use-conda

The Representational State Transfer (REST) sample clients are provided for a number of programming languages. For details of how to use these clients, download the client and run the program without any arguments.

For details see Environment setup for REST Web Services and Examples for Perl REST Web Services Clients pages.