BOL: Related items

Python Mini Projects !

BioStar — Mon, 16 Jan 2023 02:14:03 -0600

There is a directory for each chapter of the book. Each directory contains a test.py program you can use with pytest to check that you have written the program correctly. I have included a short README to describe each exercise. If you have problems writing code (or if you would like to support this project!), the book contains details about the skills you need.

https://github.com/kyclark/tiny_python_projects

Address of the bookmark: https://github.com/kyclark/tiny_python_projects

Monitor running jobs on Linux server

Jitendra Narayan — Fri, 06 Jun 2014 16:18:43 -0500

You as a bioinformatican run lots of program on your servers. Sometime the shared server is also used by your colleague. If server is busy you sometime need to check the running programs and want to monitor the running programs as well. The "top" command will come in handy when you need to find out if things are still running, how long they’ve been running, or how much memory is being used.

‘top’ is very simple to run: type

%% top

You’ll get a screen that looks like this, and is updated regularly:

Simple, right? Heh.

First! Note that you can use ‘q’ or ‘CTRL-C’ to exit from ‘top’.

Now let’s read and understand at each line independently.

The first line:

top - 23:00:48 up 39 days, 2 user, load average: 0.00, 0.00, 0.00

The first line tells you the current time, how long the machine has been up, how many users are logged in, and the short/medium/long-term compute load on the machine. If you run something for a long time, you’ll see these numbers go up. Right now, the machine is basically just sitting there, so these are all close to 0.

The second line:

Tasks: 239 total,   1 running, 238 sleeping,   0 stopped,   0 zombie

This line tells you how many processes are running. If you are using laptops machines it’s not so interesting because you really are the only one using this machine.

Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

This line contains the CPU load. The first two numbers are how busy the system is doing computation (“us” stands for “user”) and how busy the system is doing system-y things like accessing disks or network (“sy” stands for “system”). We’ll talk more about this later.

Mem:   49457320k total,    3492174k used, 14535596k free,    1435148k buffers

This should be easy to understand – how much memory you’re using!

Swap:   539356k total,   28332k used,   836562k free,    29862014k cached

Swap is just on-disk memory that can be used to “swap” out programs from main memory. Again, we’ll talk about this later.:

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
1 root      39 19 0 0 0 S 0.0 0.0   246:57.22 kipmi0
2 root      RT   0     0    0    0 S 0.0 0.0   0:00.00 migration/0

And... finally! What’s actually running! The two most important numbers are the %CPU and %MEM towards the right, as well as the COMMAND. This tells you how compute- and memory-intensive your program is. Right now, nothing’s running so the numbers aren’t very interesting, but just wait until we run something...

Vital-IT

Abhi — Thu, 18 Dec 2014 10:46:59 -0600

Vital-IT is a bioinformatics competence center that supports and collaborates with life scientists in Switzerland and beyond. The multi-disciplinary team provides expertise, training and maintains a high-performance computing (HPC) and storage infrastructure, so as to help develop, maintain and extend life science and medical research (activities).

Address of the bookmark: http://www.vital-it.ch/

SLURM basics !

Radha Agarkar — Fri, 13 May 2016 04:42:24 -0500

SLURM is a queue management system and stands for Simple Linux Utility for Resource Management. SLURM was developed at the Lawrence Livermore National Lab and currently runs some of the largest compute clusters in the world.

SLURM is similar in many ways to most other queue systems. You write a batch script then submit it to the queue manager. The queue manager then schedules your job to run on the queue (or partition in SLURM parlance) that you designate. Below we will provide an outline of how to submit jobs to SLURM, how SLURM decides when to schedule your job and how to monitor progress.

SLURM has a number of valuable features compared to other job management systems:

Kill and Requeue SLURM’s ability to kill and requeue is superior to that of other systems. It waits for jobs to be cleared before scheduling the high priority job. It also does kill and requeue on memory rather than just on core count.
Memory Memory requests are sacrosanct in SLURM. Thus the amount of memory you request at run time is guaranteed to be there. No one can infringe on that memory space and you cannot exceed the amount of memory that you request.
Accounting Tools SLURM has a back end database which stores historical information about the cluster. This information can be queried by the users who are curious about how much resources they have used.

Summary of SLURM commands

The table below shows a summary of SLURM commands. These commands are described in more detail below along with links to the SLURM doc site.

	SLURM	SLURM Example
Submit a batch serial job	sbatch	`sbatch runscript.sh`
Run a script interatively	srun	`srun --pty -p interact -t 10 --mem 1000 /bin/bash /bin/hostname`
Kill a job	scancel	`scancel 999999`
View status of queues	squeue	`squeue -u akitzmiller`
Check current job by id	sacct	`sacct -j 999999`

GWASpro: A High-Performance Genome-Wide Association Analysis Server

Rahul Nayak — Fri, 07 Dec 2018 08:04:57 -0600

GWASpro supports building complex design matrices, by which complex experimental designs that may include replications, treatments, locations and times, can be accounted for in the linear mixed model (LMM). GWASpro is optimized to handle GWAS data that may consist of up to 10 million markers and 10,000 samples from replicable lines or hybrids. GWASpro provides an interface that significantly reduces the learning curve for new GWAS investigators.

Address of the bookmark: https://bioinfo.noble.org/GWASPRO/

Stay Connected and Productive: Unlock the Power of Screen, Tmux, and Mosh for Bioinformatics

BioStar — Wed, 22 Jan 2025 00:29:52 -0600

If you are a bioinformatician, chances are you have spent hours running long, complex analyses on remote servers only to lose your session because of an unstable connection. Frustrating, isnt it? Fear not! With tools like screen, tmux, and mosh, you can safeguard your workflow and stay productive, no matter where you are.

Why Remote Session Management is a Must-Have

In bioinformatics, tasks like genome assembly, RNA-seq analyses, and phylogenetic computations often take hours or days. A dropped SSH connection can result in:

Lost Progress: Restarting a job from scratch wastes valuable time.
Workflow Interruptions: Disruptions can derail your focus and productivity.
Corrupted Data: Interrupted processes may lead to incomplete or corrupted outputs.

By integrating screen, tmux, or mosh into your workflow, you can avoid these setbacks and ensure a seamless experience.

Screen: The Classic Workhorse

Screen is a terminal multiplexer that comes pre-installed on most Linux systems. It allows you to manage multiple terminal sessions and reconnect to them even after being disconnected.

Getting Started with Screen:

Start a Session:

screen
Detach from a Session:
Press Ctrl+A, then D.
Reattach to a Session:

screen -r

Pro Tip: Enhance your screen experience with a customized .screenrc configuration file. Download one here: Get .screenrc.

Tmux: A Modern Alternative

Tmux takes everything great about screen and adds modern features, including better key bindings and intuitive session management. It\u2019s perfect for bioinformaticians who want more control over their workflow.

Getting Started with Tmux:

Start a Session:

tmux
Detach from a Session:
Press Ctrl+B, then D.
Reattach to a Session:

tmux attach

Customize Your Tmux Experience:
Use a .tmux.conf file to personalize your setup. Grab one here: Download .tmux.conf.

Mosh: The Mobile Shell for Unreliable Connections

SSH works well for stable networks, but it struggles in areas with spotty connectivity. Enter Mosh, the Mobile Shell. Designed for intermittent networks, Mosh keeps your session alive even when the connection drops temporarily.

Why Mosh is a Game-Changer:

No lag over high-latency networks.
Automatically reconnects when the network is restored.
Ideal for working on the go, from cafes to trains.

Getting Started with Mosh:

Install Mosh:

sudo apt install mosh # For Debian/Ubuntu
Connect to a Server:

mosh username@server

Learn more at mosh.org.

Why This Matters for Bioinformatics

Every bioinformatician knows the value of time and data integrity. Tools like screen, tmux, and mosh provide a lifeline when running long analyses, enabling you to:

Safeguard your work against disconnections.
Easily manage multiple workflows in parallel.
Stay productive, even in challenging environments.

Quickstart Cheat Sheet

Screen:

screen # Start a session Ctrl+A, D # Detach screen -r # Reattach
Tmux:

tmux # Start a session Ctrl+B, D # Detach tmux attach # Reattach
Mosh:

mosh username@server

Final Thoughts

As a bioinformatician, your time is too valuable to spend restarting analyses due to technical hiccups. With screen, tmux, and mosh in your toolkit, you can work smarter, protect your progress, and stay productive no matter where you are. Start using these tools today and transform the way you work with remote systems.

Let me know how these tools work for you, and don\u2019t forget to follow for more bioinformatics tips!

genomics public data links !

Jit — Thu, 13 Feb 2020 00:20:00 -0600

List of publically available databases on google server.

More at https://software.broadinstitute.org/gatk/download/bundle

ftp://ftp.ncbi.nlm.nih.gov/snp/organisms/human_9606/VCF/GATK/.

ftp://ftp.broadinstitute.org/bundle/hg38/hg38bundle/

Address of the bookmark: https://console.cloud.google.com/storage/browser/genomics-public-data/resources/broad/hg38/v0?pli=1

PPAI: a web server for predicting protein-aptamer interactions

Jit — Fri, 12 Jun 2020 07:26:23 -0500

PPAI can query aptamers and proteins, predict aptamers and predict protein-aptamer interactions in batch mode precisely and efficiently, which would be a novel bioinformatics tool for the research of protein-aptamer interactions. PPAI web-server is freely available at http://39.96.85.9/PPAI

Address of the bookmark: http://39.96.85.9/PPAI/

BioLinux Ubuntu Desktop folder and files disappeared !!

Jit — Tue, 16 Feb 2016 08:40:41 -0600

Restarted my BioLinux ubuntu computer after an update, and when I logged back in, I noticed that all of my files went missing. Instead of Desktop folder, icons of all of my home folder are showed on desktop.

Then I thaught it migh be a problem of graphical display and I opened the terminal out of curiosity, and I found out that there is no ~/Desktop folder at all. What happened? What do I need to do?

Then I google the problem and found this is a very common problem after updates. To fix this problem, follow these steps:

You need to edit the ~/.config/user-dirs.dirs file, and make sure the contents of the file are like the following:

XDG_DESKTOP_DIR="$HOME/Desktop"
XDG_DOWNLOAD_DIR="$HOME/Downloads"
XDG_TEMPLATES_DIR="$HOME/"
XDG_PUBLICSHARE_DIR="$HOME/Share"
XDG_DOCUMENTS_DIR="$HOME/Documents"
XDG_MUSIC_DIR="$HOME/Music"
XDG_PICTURES_DIR="$HOME/Pictures"
XDG_VIDEOS_DIR="$HOME/Videos"

Then restart nautilus:

killall nautilus

or

nautilus -q

Then, open nautilus via Unity menu (press the Super key) or using the run command (Alt+F2)

Pollux: platform independent error correction of single and mixed genomes

Jit — Fri, 19 May 2017 09:41:27 -0500

Pollux: General-purpose error corrector that corrects errors introduced by Illumina, Ion Torrent, and Roche 454 sequencing technologies and can be applied to single- or mixed-genome data. In addition to correcting substitution errors, we locate and correct insertion, deletion, and homopolymer errors while remaining sensitive to low coverage areas of sequencing projects. Using published data sets, we correct 94% of Illumina MiSeq errors, 88% of Ion Torrent PGM errors, 85% of Roche 454 GS Junior errors. Introduced errors are 20 to 70 times more rare than successfully corrected errors. Furthermore, we show that the quality of assemblies improves when reads are corrected by our software.

https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-014-0435-6

Address of the bookmark: https://github.com/emarinier/pollux