BOL: Related items

bioinformatics workbook

biogeek — Tue, 05 Jan 2021 22:42:32 -0600

This books assumes that the reader has some knowledge of biology and basic understanding of the Unix command line. However, for the beginner, the appendix contains introductory material and tips/tricks for common bioinformatic problems, that is referred to for more information throughout the book.

https://bioinformaticsworkbook.org/

Address of the bookmark: https://bioinformaticsworkbook.org/

biostarhandbook

Neel — Fri, 27 Aug 2021 01:31:01 -0500

Nice book collection for bioinformatician ... highly recommended.

Address of the bookmark: https://www.biostarhandbook.com/

Python Mini Projects !

BioStar — Mon, 16 Jan 2023 02:14:03 -0600

There is a directory for each chapter of the book. Each directory contains a test.py program you can use with pytest to check that you have written the program correctly. I have included a short README to describe each exercise. If you have problems writing code (or if you would like to support this project!), the book contains details about the skills you need.

https://github.com/kyclark/tiny_python_projects

Address of the bookmark: https://github.com/kyclark/tiny_python_projects

Monitor running jobs on Linux server

Jitendra Narayan — Fri, 06 Jun 2014 16:18:43 -0500

You as a bioinformatican run lots of program on your servers. Sometime the shared server is also used by your colleague. If server is busy you sometime need to check the running programs and want to monitor the running programs as well. The "top" command will come in handy when you need to find out if things are still running, how long they’ve been running, or how much memory is being used.

‘top’ is very simple to run: type

%% top

You’ll get a screen that looks like this, and is updated regularly:

Simple, right? Heh.

First! Note that you can use ‘q’ or ‘CTRL-C’ to exit from ‘top’.

Now let’s read and understand at each line independently.

The first line:

top - 23:00:48 up 39 days, 2 user, load average: 0.00, 0.00, 0.00

The first line tells you the current time, how long the machine has been up, how many users are logged in, and the short/medium/long-term compute load on the machine. If you run something for a long time, you’ll see these numbers go up. Right now, the machine is basically just sitting there, so these are all close to 0.

The second line:

Tasks: 239 total,   1 running, 238 sleeping,   0 stopped,   0 zombie

This line tells you how many processes are running. If you are using laptops machines it’s not so interesting because you really are the only one using this machine.

Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

This line contains the CPU load. The first two numbers are how busy the system is doing computation (“us” stands for “user”) and how busy the system is doing system-y things like accessing disks or network (“sy” stands for “system”). We’ll talk more about this later.

Mem:   49457320k total,    3492174k used, 14535596k free,    1435148k buffers

This should be easy to understand – how much memory you’re using!

Swap:   539356k total,   28332k used,   836562k free,    29862014k cached

Swap is just on-disk memory that can be used to “swap” out programs from main memory. Again, we’ll talk about this later.:

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
1 root      39 19 0 0 0 S 0.0 0.0   246:57.22 kipmi0
2 root      RT   0     0    0    0 S 0.0 0.0   0:00.00 migration/0

And... finally! What’s actually running! The two most important numbers are the %CPU and %MEM towards the right, as well as the COMMAND. This tells you how compute- and memory-intensive your program is. Right now, nothing’s running so the numbers aren’t very interesting, but just wait until we run something...

Google Genomics

Tenzin Paul — Thu, 18 Dec 2014 11:05:42 -0600

Explore genetic variation interactively. Compare entire cohorts in seconds with SQL-like queries. Compute transition/transversion ratios, genome-wide association, allelic frequency and more.
Process big genomic data easily. Run batch analyses like principal component analysis and Hardy-Weinberg equilibrium on as many samples as you like, in minutes or hours, with just a little code.
Use Google's infrastructure and big data expertise. Store one genome or a million using Google Genomics and take advantage of the same infrastructure that powers Search, Maps, YouTube, Gmail and Drive.
Support emerging global standards. Google Genomics is implementing the API defined by the Global Alliance for Genomics and Health for visualization, analysis and more. Compliant software can access Google Genomics, local servers, or any other implementation.

Address of the bookmark: https://cloud.google.com/genomics/

Enrichr: a comprehensive gene set enrichment analysis

Jit — Thu, 27 Apr 2017 05:42:09 -0500

Enrichment analysis is a popular method for analyzing gene sets generated by genome-wide experiments. Here we present a significant update to one of the tools in this domain called Enrichr. Enrichr currently contains a large collection of diverse gene set libraries available for analysis and download. In total, Enrichr currently contains 180 184 annotated gene sets from 102 gene set libraries. New features have been added to Enrichr including the ability to submit fuzzy sets, upload BED files, improved application programming interface and visualization of the results as clustergrams. Overall, Enrichr is a comprehensive resource for curated gene sets and a search engine that accumulates biological knowledge for further biological discoveries. Enrichr is freely available at: http://amp.pharm.mssm.edu/Enrichr.

https://academic.oup.com/nar/article-lookup/doi/10.1093/nar/gkw377

Address of the bookmark: http://amp.pharm.mssm.edu/Enrichr/

Installing ELGG on Ubuntu !

Jit — Thu, 25 Nov 2021 01:45:59 -0600

Follow this:

Settings file

Error

Your web server does not have permission to create the settings.php file in your installation directory. You have two choices:

1. Change the permissions on the elgg-config directory of your Elgg installation

2. Copy the file elgg-config/settings.example.php to elgg-config/settings.php and follow the instructions in it for setting your database parameters.

To resolve this --

Change the permision

chmod 777 elgg-config

Web server

Warning

Your server does not support automatic testing of the rewrite rules and your browser does not support checking via JavaScript. You can continue the installation, but you may experience problems with your site. You can manually test the rewrite rules by clicking this link: test. You will see the word success if the rules are working.

create a .httaccess

and provide detail in it

Installation of Elgg

(Note: Elgg requires installing an instance of MySQL.)

Steps to make adjustment and disposal of trial installations easier.

MySQL

At this point, we assume you've set up your MySQL database

Start your mysqld server daemon.
(For the sake of this example, we'll assume we're running on port 9999; you shouldn't actually use this port because only one daemon can run on any port number.)
Login to the server as root user, using the mysql client.
Create a database to store Elgg's tables. We'll call the database elggalpha.
mysql> create database elggalpha;

Grant access to a user — for this example we'll call him/her elggalphauser.

mysql> grant all privileges on elggalpha.* to 'elggalphauser'@'moxie'
    -> identified by 'secretpassword';

Elgg

Type umask 022
Change into your public_html directory (aka folder).
cd public_html
make a directory called elgg
mkdir elgg
Change into the elgg subdirectory of public_html.
cd elgg
download elgg into this directory
you'll end up with a file named
/home/your-user-id/public_html/elgg/elgg-X.X.X.X.zip
(It's not literally X.X.X.X, it might be elgg-1.8.0.1.zip, for example.)
unzip elgg
unzip -q elgg-1.8.0.1.zip
you'll end up with a directory (folder) named
/home/your-user-id/public_html/elgg/elgg-X.X.X.X
make a symbolic link (aka shortcut) to this
ln -s elgg-X.X.X.X alpha
(For example, ln -s elgg-1.8.0.1 alpha)
Give user nobody access to this directory
/usr/misc/bin/acl.sh -r -u nobody elgg-1.8.0.1
create a data directory for elgg (get into your home directory first):
```
cd
mkdir -p elgg/alpha/data
chmod -R 755 elgg/alpha/data
```
Change into the elgg/alpha directory.
cd elgg/alpha
Give user nobody access to this subdirectory data
/usr/misc/bin/acl.sh -u nobody data

In your browser, navigate to your elgg installation
http://moxie.cs.oswego.edu/~your-user-id/elgg/alpha/
You'll have a screen which includes this message:

Welcome

Installing Elgg has 6 simple steps and reading this welcome is the first one!

If you haven't already, read through the installation instructions included with Elgg (or click the instructions link at the bottom of the page).

If you are ready to proceed, click the Next button.

Follow configuration process:

The first screen probably will contain a success message in green and a failure message in pink:
```
Requirements check

Your server failed the requirements check. After you have fixed the below issues, refresh this
page. Check the troubleshooting links at the bottom of this page if you need further assistance.
PHP
```
Your server's PHP satisfies all of Elgg's requirements.
```
Web server
```
We think your server is running the Apache web server. The rewrite test failed and the most likely cause is that AllowOverride is not set to All for Elgg's directory. This prevents Apache from processing the .htaccess file which contains the rewrite rules. A less likely cause is Apache is configured with an alias for your Elgg directory and you need to set the RewriteBase in your .htaccess. There are further instructions in the .htaccess file in your Elgg directory.
```
Database
```
The database requirements are checked when Elgg loads its database.
To correct this, replace the .htaccess file that Elgg created with one that you can edit:
```
cp -ip .htaccess temp-htaccess
rm .htaccess
mv temp-htaccess .htaccess
```
Edit the .htaccess file: Go to approximately line 101 (where it says #RewriteBase /) and add the line:
RewriteBase /~your-user-id/elgg/alpha/
Be sure to save the edited file.
Click the Refresh button.
If this hasn't fixed things, seek professional help.
Click the Next button.

Fill in the parameters appropriate to your MySQL installation.

Database installation

If you haven't already created a database for Elgg, do that now. Then fill in the values below to
initialize the Elgg database.

Database Username

elggalphauser

User that has full priviledges to the MySQL database that you created for Elgg

Database Password

secretpassword

Password for the above database user account

Database Name

elggalpha

Name of the Elgg database

Database Host

^‡

moxie:9999

Hostname of the MySQL server (usually localhost)
‡You might need to use 127.0.0.1:9999 for the host
    if you haven't set up MySQL as we've done.



Database Table Prefix

elgg_

The prefix given to all of Elgg's tables (usually elgg_)

Continue filling in forms as requested.

Configure site

Database has been installed.

We need some information about the site as we configure Elgg. If you haven't created a data directory for Elgg, you need to do so now.

Site Name

My Elgg Site version Alpha

The name of your new Elgg site

Site Email Address

your-user-id@oswego.edu

Email address used by Elgg for communication with users

Site URL

http://moxie.cs.oswego.edu/~your-user-id/elgg/alpha/

The address of the site (Elgg usually guesses this correctly)

Elgg Install Directory

/home/your-user-id/public_html/elgg/elgg.1.8.0.1/

The directory where you put the Elgg code (Elgg usually guesses this correctly)

Data Directory

/home/your-user-id/elgg/alpha/data

The directory that you created for Elgg to save files (the permissions on this directory are checked
when you click Next)

Default Site Access

Public

The default access level for new user created content

Create admin account

Site settings have been saved.

It is now time to create an administrator's account.

Display Name

The name that is displayed on the site for this account

Email Address

Username

Account username used for logging in

Password

Account password must be at least 6 characters long

Password Again

Retype password to confirm

Stay Connected and Productive: Unlock the Power of Screen, Tmux, and Mosh for Bioinformatics

BioStar — Wed, 22 Jan 2025 00:29:52 -0600

If you are a bioinformatician, chances are you have spent hours running long, complex analyses on remote servers only to lose your session because of an unstable connection. Frustrating, isnt it? Fear not! With tools like screen, tmux, and mosh, you can safeguard your workflow and stay productive, no matter where you are.

Why Remote Session Management is a Must-Have

In bioinformatics, tasks like genome assembly, RNA-seq analyses, and phylogenetic computations often take hours or days. A dropped SSH connection can result in:

Lost Progress: Restarting a job from scratch wastes valuable time.
Workflow Interruptions: Disruptions can derail your focus and productivity.
Corrupted Data: Interrupted processes may lead to incomplete or corrupted outputs.

By integrating screen, tmux, or mosh into your workflow, you can avoid these setbacks and ensure a seamless experience.

Screen: The Classic Workhorse

Screen is a terminal multiplexer that comes pre-installed on most Linux systems. It allows you to manage multiple terminal sessions and reconnect to them even after being disconnected.

Getting Started with Screen:

Start a Session:

screen
Detach from a Session:
Press Ctrl+A, then D.
Reattach to a Session:

screen -r

Pro Tip: Enhance your screen experience with a customized .screenrc configuration file. Download one here: Get .screenrc.

Tmux: A Modern Alternative

Tmux takes everything great about screen and adds modern features, including better key bindings and intuitive session management. It\u2019s perfect for bioinformaticians who want more control over their workflow.

Getting Started with Tmux:

Start a Session:

tmux
Detach from a Session:
Press Ctrl+B, then D.
Reattach to a Session:

tmux attach

Customize Your Tmux Experience:
Use a .tmux.conf file to personalize your setup. Grab one here: Download .tmux.conf.

Mosh: The Mobile Shell for Unreliable Connections

SSH works well for stable networks, but it struggles in areas with spotty connectivity. Enter Mosh, the Mobile Shell. Designed for intermittent networks, Mosh keeps your session alive even when the connection drops temporarily.

Why Mosh is a Game-Changer:

No lag over high-latency networks.
Automatically reconnects when the network is restored.
Ideal for working on the go, from cafes to trains.

Getting Started with Mosh:

Install Mosh:

sudo apt install mosh # For Debian/Ubuntu
Connect to a Server:

mosh username@server

Learn more at mosh.org.

Why This Matters for Bioinformatics

Every bioinformatician knows the value of time and data integrity. Tools like screen, tmux, and mosh provide a lifeline when running long analyses, enabling you to:

Safeguard your work against disconnections.
Easily manage multiple workflows in parallel.
Stay productive, even in challenging environments.

Quickstart Cheat Sheet

Screen:

screen # Start a session Ctrl+A, D # Detach screen -r # Reattach
Tmux:

tmux # Start a session Ctrl+B, D # Detach tmux attach # Reattach
Mosh:

mosh username@server

Final Thoughts

As a bioinformatician, your time is too valuable to spend restarting analyses due to technical hiccups. With screen, tmux, and mosh in your toolkit, you can work smarter, protect your progress, and stay productive no matter where you are. Start using these tools today and transform the way you work with remote systems.

Let me know how these tools work for you, and don\u2019t forget to follow for more bioinformatics tips!

How to set up SSH on Ubuntu 18.04

AnHo — Thu, 22 Nov 2018 10:12:00 -0600

SSH, also known as Secure Shell or Secure Socket Shell, is a network protocol that gives users, particularly system administrators, a secure way to access a computer over an unsecured network. SSH also refers to the suite of utilities that implement the SSH protocol.

Here are the commands used to connect by Secure SHell:

On the server side

sudo apt-get install ssh
sudo apt-get install openssh-server
sudo /etc/init.d/ssh start
sudo nano /etc/ssh/sshd_config
Uncomment port 22
Uncomment HostKey /etc/ssh/ssh_host_rsa_key
Uncomment AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
Set pubkey authentication to "yes"

sudo systemctl restart sshd.service # or sudo /etc/init.d/ssh reload

On the client side:
in ~/.ssh

ssh-keygen -t rsa # set passphrase or not
ssh-copy-id -i ~/.ssh/id_rsa user@100.100.10.100

--> write "yes" then password in

ssh user@100.100.10.100

--> write password --> you should be in

--> in /home/user/.ssh/config type:
Host WhateverName
HostName 100.100.10.100
User username
ForwardX11 yes
ForwardAgent yes
IdentityFile ~/.ssh/id_rsa

--> you should now be able to connect with :

ssh WhateverName

genomics public data links !

Jit — Thu, 13 Feb 2020 00:20:00 -0600

List of publically available databases on google server.

More at https://software.broadinstitute.org/gatk/download/bundle

ftp://ftp.ncbi.nlm.nih.gov/snp/organisms/human_9606/VCF/GATK/.

ftp://ftp.broadinstitute.org/bundle/hg38/hg38bundle/

Address of the bookmark: https://console.cloud.google.com/storage/browser/genomics-public-data/resources/broad/hg38/v0?pli=1