BOL: Related items

New BLAST Core Nucleotide Database (core_nt)

LEGE — Tue, 13 Aug 2024 07:12:53 -0500

The Core Nucleotide Database (core_nt) is now the default nucleotide BLAST database. Core_nt is also available on the command line. You get faster searches & more focused results.

Core_nt contains the same eukaryotic transcript and gene-related sequences as nt. The core_nt database is nt without most eukaryotic chromosome sequences. Most nucleotide BLAST searches with core_nt will be similar to the nt database. However, core_nt is better than nt for accomplishing your most common BLAST search goals, such as identifying gene-related sequences like transcript sequences and complete bacterial chromosomes. This is because, in recent years, nt has acquired more low-relevance, non-annotated, and non-gene content.

Learn more: https://ncbiinsights.ncbi.nlm.nih.gov/2024/07/18/new-blast-core-nucleotide-database/

Public Databases for Bioinformatics !

Jit — Tue, 23 Mar 2021 05:32:15 -0500

https://www.nature.com/articles/s41467-020-17155-y

Server Infrastructure:

File Server:

dhara: Synology 3614 Storage Appliance
4 Core Xeon
108TB disk storage
10Gb ethernet to SCG3
Access atx: dhara:5000
Has btsync server (try it - its much better than dropbox)

Compute Servers:

nandi: Kundaje and Phi Server
24 intel cores
256GB RAM
500GB of SSD storage 
36TB RAID6 local storage
4 Intel Phi's (space for 4 more GPU's)


durga: Montgomery and sensitive data
24 intel cores
256GB RAM
500GB of SSD RAID0 storage 
60TB RAID6 local storage

mitra: Bassik and Web/DB Server
24 core
256GB RAM 
500GB of SSD RAID0 storage 
36TB RAID6 local storage

vayu: Kundaje GPU server
4 core
64GB RAM 
200GB of SSD storage 
8TB RAID10 local storage
4 Nvidia GTX 970 4GB GPUs

amold: Bickel and SGE server
32 AMD core
128GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

wotan: Bickel and SGE server
64 AMD core
256GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

Filesystem:

/users/$USER
default home directory
full backups nightly 
nfs mount to dhara
should store code, papers, and other highly processed data here

/mnt/data/
globally accessible data
should store common data here
e.g. genomes and indexes, annotations, ENCODE data  
if you dont want this to count towards your quote you must chown

/mnt/lab_data/$LAB/
lab accessible data
should store lab project data here 
e.g. ATAC-seq prediction data, enhancer prediction, motif calls

/srv/scratch/$USER
fast local storage
not backed up, but on raid and data will never be deleted
most analysis should be performed here

/srv/persistent/$USER
fast local storage
synced nightly, but not backed up
       ie if the hard drives fail or you delete something and notice 
       within 24 hours we can recover. Otherwise not. (vs home which is 
       properly backed up )  
intermediate analysis products that would be hard to recover should be stored here 
       e.g. stochastic analysis results that need to be kept so that paper 
       results can be reproduced

/srv/www/$LABNAME/
web accessible from mitra.stanford.edu
*NOT BACKED UP*

Some parallel programming patterns:

# gzip a bunch of files
parallel gzip -- *.FILESTOGZIP

# fork example in python:
(for more detailed examples look at 
 https://github.com/nboley/grit/ grit/lib/multiprocessing_utils.py)

import os
import time
import random

import multiprocessing

class ProcessSafeOPStream( object ):
    def __init__( self, writeable_obj ):
        self.writeable_obj = writeable_obj
        self.lock = multiprocessing.Lock()
        self.name = self.writeable_obj.name
        return
    
    def write( self, data ):
        self.lock.acquire()
        self.writeable_obj.write( data )
        self.writeable_obj.flush()
        self.lock.release()
        return
    
    def close( self ):
        self.writeable_obj.close()

def worker(queue, ofp):
    # Try without this
    random.seed()
    while True:
        i = queue.get()
        if i == 'FINISHED': return
        # simulate an expensive function
        x = random.random()
        time.sleep(x/10)
        print i, x
        ofp.write("%i\t%s\n" % (i, x))

NSIMS = 10000
NPROC = 25

# populate queue
todo = multiprocessing.Queue()
for i in xrange(NSIMS): todo.put(i)
for i in xrange(NPROC): todo.put('FINISHED')

ofp = ProcessSafeOPStream( open("output.txt", "w") )

pids = []
for i in xrange(NPROC):
    pid = os.fork()
    if pid == 0:
       worker(todo, ofp)
       os._exit(0)
    else:
       pids.append(pid)  

for pid in pids:
    os.waitpid(pid, 0)

ofp.close()

print "FINISHED"

For use case 1 we obtained the following ENCODE and ROADMAP datasets https://www.encodeproject.org/files/ENCFF446WOD/@@download/ENCFF446WOD.bed.gz, https://www.encodeproject.org/files/ENCFF546PJU/@@download/ENCFF546PJU.bam, https://www.encodeproject.org/files/ENCFF059BEU/@@download/ENCFF059BEU.bam. Blacklisted regions were obtained from http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/hg38-human/hg38.blacklist.bed.gz. The human genome version hg38 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz.

For use case 2 we used the set of narrowPeak files summarized in https://github.com/wkopp/janggu_usecases/tree/master/extra/urls.txt (archived version v1.0.1). The human genome version hg19 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/hg19.fa.gz

For use case 3 we used the ENCODE datasets https://www.encodeproject.org/files/ENCFF591XCX/@@download/ENCFF591XCX.bam, https://www.encodeproject.org/files/ENCFF736LHE/@@download/ENCFF736LHE.bigWig, https://www.encodeproject.org/files/ENCFF177HHM/@@download/ENCFF177HHM.bam as we as the GENCODE annotation v29 from ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_29/gencode.v29.annotation.gtf.gz.

Address of the bookmark: http://mitra.stanford.edu/

BLAST nr version 5 database, (nr_v5)

Jit — Fri, 23 Aug 2019 11:35:35 -0500

NCBI have made changes the nr version 5 database, (nr_v5), to facilitate better search results and improved performance by reducing the number of redundant titles in the nr_v5 database used by webBLAST, which is also available for BLAST+ users.

The changes in nr preserve the taxonomic diversity of the entries in the database while reducing the number of titles for identical sequences. GenPept accessions are still accessible via www.ncbi.nlm.nih.gov/protein/$GENBANK_ACCESSION or the IPG website https://www.ncbi.nlm.nih.gov/ipg/.

The "Identical Proteins" link in the alignments section of the webBLAST results takes you to a full list of all accessions associated with a sequence.

For BLAST+ users downloading nr_v5: the database is now approximately 50% smaller, resulting in faster downloads and BLAST searches, and smaller disk space requirements. The database is downloadable at: ftp://ftp.ncbi.nlm.nih.gov/blast/db/v5/

For BLAST+ there is a cleanup script to help you manage the transition to this smaller database. The script removes unused database volumes: ftp://ftp.ncbi.nlm.nih.gov/blast/temp/cleanup-blastdb-volumes.py

Here are the new rules on how we keep titles in nr_v5:

1. We keep all refseq, swissprot, pir and PDB titles.

2. We keep any GenPept titles with a TAXID that has not already been seen in the record.

3. We keep at least five GenPept titles regardless of whether the TAXIDS have been seen before or not in this record.

DEG 5.0: a database of essential genes in both prokaryotes and eukaryotes

Rahul Nayak — Tue, 30 Mar 2021 11:47:28 -0500

Essential genes are those indispensable for the survival of an organism, and their functions are therefore considered a foundation of life. Determination of a minimal gene set needed to sustain a life form, a fundamental question in biology, plays a key role in the emerging field, synthetic biology.

DEG is freely available at the website http://tubic.tju.edu.cn/deg or http://www.essentialgene.org.

Address of the bookmark: http://www.essentialgene.org/

ChIPBase: open database for studying the transcription factor binding sites and motifs

Abhi — Wed, 29 Dec 2021 05:36:03 -0600

ChIPBase v2.0 is an open database for studying the transcription factor binding sites and motifs, and decoding the transcriptional regulatory networks of lncRNAs, miRNAs, other ncRNAs and protein-coding genes from ChIP-seq data. Our database currently contains ~10,200 curated peak datasets derived from ChIP-seq methods in 10 species.

Address of the bookmark: https://rna.sysu.edu.cn/chipbase/

geNomad: Identification of mobile genetic elements

LEGE — Sun, 31 Aug 2025 06:40:17 -0500

geNomad is a tool that identifies virus and plasmid genomes from nucleotide sequences. It provides state-of-the-art classification performance and can be used to quickly find mobile genetic elements from genomes, metagenomes, or metatranscriptomes.

Address of the bookmark: https://portal.nersc.gov/genomad

McClintock: Meta-pipeline to identify transposable element insertions using next generation sequencing data

BioStar — Tue, 27 Oct 2020 00:21:18 -0500

an integrated bioinformatics pipeline for the detection of TE insertions in whole-genome shotgun data, called McClintock (https://github.com/bergmanlab/mcclintock), which automatically runs and standardizes output for multiple TE detection methods. We demonstrate the utility of McClintock by evaluating six TE detection methods using simulated and real genome data from the model microbial eukaryote, Saccharomyces cerevisiae.

Address of the bookmark: https://github.com/bergmanlab/mcclintock

Cancer research database

Neel — Tue, 01 Sep 2015 17:36:31 -0500

Researchers in Andhra Pradesh have developed a database to identify genes that are common in tumours to provide their colleagues with easy access to insights into the genetic alterations in cancer.

The database, hosted at the Sri Venkateswara University (SVU) in Tirupati, will integrate information on cancer genes and markers with experimental data.

The Cancer Gene Markers Database (CGMD) is meant to help scientists better understand tumour genes and markers at a molecular level by combining data with literature on treatment regimen and recent advances in cancer therapy.

The database is free to access, and already includes 309 genes and 206 markers that correspond to 40 different human cancers. Accompanying literature comes from databases such as the United States’ National Center for Biotechnology Information and the Kyoto Encyclopedia of Genes and Genomes. It also includes experimental data from PubMed.

In a paper published last month in Nature Scientific Reports, the researchers from SVU’s department of animal biotechnology, describes the need for a database for different genes and markers along with their molecular characteristics and pathway associations.

Gene Synteny Database

Jit — Fri, 16 Dec 2016 11:09:39 -0600

Comparative genomics remains a pivotal strategy to study the evolution of gene organization, and this primacy is reinforced by the growing number of full genome sequences available in public repositories. Despite this growth, bioinformatic tools available to visualize and compare genomes and to infer evolutionary events remain restricted to two or three genomes at a time, thus limiting the breadth and the nature of the question that can be investigated. Here we present Genomicus, a new synteny browser that can represent and compare unlimited numbers of genomes in a broad phylogenetic view. In addition, Genomicus includes reconstructed ancestral gene organization, thus greatly facilitating the interpretation of the data.

Availability: Genomicus is freely available for online use at http://www.dyogen.ens.fr/genomicus while data can be downloaded at ftp://ftp.biologie.ens.fr/pub/dyogen/genomicus

Contact: rf.sne.eigoloib@crh

Address of the bookmark: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2853686/

New Layout for BLAST ftp Database Site

Jit — Tue, 21 Jan 2020 11:57:11 -0600

As announced previously, the new default database version for BLAST+ is dbV5. To complete this transition, the ftp database site will be updated to support this change. We expect this change to happen around February 4^th, please adjust your scripts or procedures accordingly.

Here is a list of what is changing:

All databases at the root level will be dbV5.
The dbV5 file naming, “_v5” will be removed. Databases with no “_vX” descriptor will be dbV5.
dbV4 tarballs will be renamed with "_v4", files included in tarball will not be renamed.
dbV4 databases will be moved to a v4 subdirectory.
As of 1/13/20 the Cloud directory will be frozen with no more new entries.
The will be no more updates to dbV4 databases.
The FASTA directory will contain nr, nt, swissprot, and pdbaa files.

If you have any questions or concerns, please contact blast-help@ncbi.nlm.nih.gov