BOL: Related items

Public Databases for Bioinformatics !

Jit — Tue, 23 Mar 2021 05:32:15 -0500

https://www.nature.com/articles/s41467-020-17155-y

Server Infrastructure:

File Server:

dhara: Synology 3614 Storage Appliance
4 Core Xeon
108TB disk storage
10Gb ethernet to SCG3
Access atx: dhara:5000
Has btsync server (try it - its much better than dropbox)

Compute Servers:

nandi: Kundaje and Phi Server
24 intel cores
256GB RAM
500GB of SSD storage 
36TB RAID6 local storage
4 Intel Phi's (space for 4 more GPU's)


durga: Montgomery and sensitive data
24 intel cores
256GB RAM
500GB of SSD RAID0 storage 
60TB RAID6 local storage

mitra: Bassik and Web/DB Server
24 core
256GB RAM 
500GB of SSD RAID0 storage 
36TB RAID6 local storage

vayu: Kundaje GPU server
4 core
64GB RAM 
200GB of SSD storage 
8TB RAID10 local storage
4 Nvidia GTX 970 4GB GPUs

amold: Bickel and SGE server
32 AMD core
128GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

wotan: Bickel and SGE server
64 AMD core
256GB RAM 
200GB of SSD storage 
12TB RAID5 local storage

Filesystem:

/users/$USER
default home directory
full backups nightly 
nfs mount to dhara
should store code, papers, and other highly processed data here

/mnt/data/
globally accessible data
should store common data here
e.g. genomes and indexes, annotations, ENCODE data  
if you dont want this to count towards your quote you must chown

/mnt/lab_data/$LAB/
lab accessible data
should store lab project data here 
e.g. ATAC-seq prediction data, enhancer prediction, motif calls

/srv/scratch/$USER
fast local storage
not backed up, but on raid and data will never be deleted
most analysis should be performed here

/srv/persistent/$USER
fast local storage
synced nightly, but not backed up
       ie if the hard drives fail or you delete something and notice 
       within 24 hours we can recover. Otherwise not. (vs home which is 
       properly backed up )  
intermediate analysis products that would be hard to recover should be stored here 
       e.g. stochastic analysis results that need to be kept so that paper 
       results can be reproduced

/srv/www/$LABNAME/
web accessible from mitra.stanford.edu
*NOT BACKED UP*

Some parallel programming patterns:

# gzip a bunch of files
parallel gzip -- *.FILESTOGZIP

# fork example in python:
(for more detailed examples look at 
 https://github.com/nboley/grit/ grit/lib/multiprocessing_utils.py)

import os
import time
import random

import multiprocessing

class ProcessSafeOPStream( object ):
    def __init__( self, writeable_obj ):
        self.writeable_obj = writeable_obj
        self.lock = multiprocessing.Lock()
        self.name = self.writeable_obj.name
        return
    
    def write( self, data ):
        self.lock.acquire()
        self.writeable_obj.write( data )
        self.writeable_obj.flush()
        self.lock.release()
        return
    
    def close( self ):
        self.writeable_obj.close()

def worker(queue, ofp):
    # Try without this
    random.seed()
    while True:
        i = queue.get()
        if i == 'FINISHED': return
        # simulate an expensive function
        x = random.random()
        time.sleep(x/10)
        print i, x
        ofp.write("%i\t%s\n" % (i, x))

NSIMS = 10000
NPROC = 25

# populate queue
todo = multiprocessing.Queue()
for i in xrange(NSIMS): todo.put(i)
for i in xrange(NPROC): todo.put('FINISHED')

ofp = ProcessSafeOPStream( open("output.txt", "w") )

pids = []
for i in xrange(NPROC):
    pid = os.fork()
    if pid == 0:
       worker(todo, ofp)
       os._exit(0)
    else:
       pids.append(pid)  

for pid in pids:
    os.waitpid(pid, 0)

ofp.close()

print "FINISHED"

For use case 1 we obtained the following ENCODE and ROADMAP datasets https://www.encodeproject.org/files/ENCFF446WOD/@@download/ENCFF446WOD.bed.gz, https://www.encodeproject.org/files/ENCFF546PJU/@@download/ENCFF546PJU.bam, https://www.encodeproject.org/files/ENCFF059BEU/@@download/ENCFF059BEU.bam. Blacklisted regions were obtained from http://mitra.stanford.edu/kundaje/akundaje/release/blacklists/hg38-human/hg38.blacklist.bed.gz. The human genome version hg38 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz.

For use case 2 we used the set of narrowPeak files summarized in https://github.com/wkopp/janggu_usecases/tree/master/extra/urls.txt (archived version v1.0.1). The human genome version hg19 was obtained from http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/hg19.fa.gz

For use case 3 we used the ENCODE datasets https://www.encodeproject.org/files/ENCFF591XCX/@@download/ENCFF591XCX.bam, https://www.encodeproject.org/files/ENCFF736LHE/@@download/ENCFF736LHE.bigWig, https://www.encodeproject.org/files/ENCFF177HHM/@@download/ENCFF177HHM.bam as we as the GENCODE annotation v29 from ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_29/gencode.v29.annotation.gtf.gz.

Address of the bookmark: http://mitra.stanford.edu/

BLAST nr version 5 database, (nr_v5)

Jit — Fri, 23 Aug 2019 11:35:35 -0500

NCBI have made changes the nr version 5 database, (nr_v5), to facilitate better search results and improved performance by reducing the number of redundant titles in the nr_v5 database used by webBLAST, which is also available for BLAST+ users.

The changes in nr preserve the taxonomic diversity of the entries in the database while reducing the number of titles for identical sequences. GenPept accessions are still accessible via www.ncbi.nlm.nih.gov/protein/$GENBANK_ACCESSION or the IPG website https://www.ncbi.nlm.nih.gov/ipg/.

The "Identical Proteins" link in the alignments section of the webBLAST results takes you to a full list of all accessions associated with a sequence.

For BLAST+ users downloading nr_v5: the database is now approximately 50% smaller, resulting in faster downloads and BLAST searches, and smaller disk space requirements. The database is downloadable at: ftp://ftp.ncbi.nlm.nih.gov/blast/db/v5/

For BLAST+ there is a cleanup script to help you manage the transition to this smaller database. The script removes unused database volumes: ftp://ftp.ncbi.nlm.nih.gov/blast/temp/cleanup-blastdb-volumes.py

Here are the new rules on how we keep titles in nr_v5:

1. We keep all refseq, swissprot, pir and PDB titles.

2. We keep any GenPept titles with a TAXID that has not already been seen in the record.

3. We keep at least five GenPept titles regardless of whether the TAXIDS have been seen before or not in this record.

DEG 5.0: a database of essential genes in both prokaryotes and eukaryotes

Rahul Nayak — Tue, 30 Mar 2021 11:47:28 -0500

Essential genes are those indispensable for the survival of an organism, and their functions are therefore considered a foundation of life. Determination of a minimal gene set needed to sustain a life form, a fundamental question in biology, plays a key role in the emerging field, synthetic biology.

DEG is freely available at the website http://tubic.tju.edu.cn/deg or http://www.essentialgene.org.

Address of the bookmark: http://www.essentialgene.org/

ChIPBase: open database for studying the transcription factor binding sites and motifs

Abhi — Wed, 29 Dec 2021 05:36:03 -0600

ChIPBase v2.0 is an open database for studying the transcription factor binding sites and motifs, and decoding the transcriptional regulatory networks of lncRNAs, miRNAs, other ncRNAs and protein-coding genes from ChIP-seq data. Our database currently contains ~10,200 curated peak datasets derived from ChIP-seq methods in 10 species.

Address of the bookmark: https://rna.sysu.edu.cn/chipbase/

geNomad: Identification of mobile genetic elements

LEGE — Sun, 31 Aug 2025 06:40:17 -0500

geNomad is a tool that identifies virus and plasmid genomes from nucleotide sequences. It provides state-of-the-art classification performance and can be used to quickly find mobile genetic elements from genomes, metagenomes, or metatranscriptomes.

Address of the bookmark: https://portal.nersc.gov/genomad

RepeatModeler2: automated genomic discovery of transposable element families

Neel — Mon, 02 Dec 2019 06:52:29 -0600

RepeatModeler2 represents a valuable addition to the genome annotation toolkit that will enhance the identification and study of TEs in eukaryotic genome sequences. RepeatModeler2 is available as source code or a containerized package under an open license (https://github.com/Dfam-consortium/RepeatModeler, https://github.com/Dfam-consortium/TETools).

Address of the bookmark: https://github.com/Dfam-consortium/TETools

Cancer research database

Neel — Tue, 01 Sep 2015 17:36:31 -0500

Researchers in Andhra Pradesh have developed a database to identify genes that are common in tumours to provide their colleagues with easy access to insights into the genetic alterations in cancer.

The database, hosted at the Sri Venkateswara University (SVU) in Tirupati, will integrate information on cancer genes and markers with experimental data.

The Cancer Gene Markers Database (CGMD) is meant to help scientists better understand tumour genes and markers at a molecular level by combining data with literature on treatment regimen and recent advances in cancer therapy.

The database is free to access, and already includes 309 genes and 206 markers that correspond to 40 different human cancers. Accompanying literature comes from databases such as the United States’ National Center for Biotechnology Information and the Kyoto Encyclopedia of Genes and Genomes. It also includes experimental data from PubMed.

In a paper published last month in Nature Scientific Reports, the researchers from SVU’s department of animal biotechnology, describes the need for a database for different genes and markers along with their molecular characteristics and pathway associations.

Gene Synteny Database

Jit — Fri, 16 Dec 2016 11:09:39 -0600

Comparative genomics remains a pivotal strategy to study the evolution of gene organization, and this primacy is reinforced by the growing number of full genome sequences available in public repositories. Despite this growth, bioinformatic tools available to visualize and compare genomes and to infer evolutionary events remain restricted to two or three genomes at a time, thus limiting the breadth and the nature of the question that can be investigated. Here we present Genomicus, a new synteny browser that can represent and compare unlimited numbers of genomes in a broad phylogenetic view. In addition, Genomicus includes reconstructed ancestral gene organization, thus greatly facilitating the interpretation of the data.

Availability: Genomicus is freely available for online use at http://www.dyogen.ens.fr/genomicus while data can be downloaded at ftp://ftp.biologie.ens.fr/pub/dyogen/genomicus

Contact: rf.sne.eigoloib@crh

Address of the bookmark: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2853686/

New Layout for BLAST ftp Database Site

Jit — Tue, 21 Jan 2020 11:57:11 -0600

As announced previously, the new default database version for BLAST+ is dbV5. To complete this transition, the ftp database site will be updated to support this change. We expect this change to happen around February 4^th, please adjust your scripts or procedures accordingly.

Here is a list of what is changing:

All databases at the root level will be dbV5.
The dbV5 file naming, “_v5” will be removed. Databases with no “_vX” descriptor will be dbV5.
dbV4 tarballs will be renamed with "_v4", files included in tarball will not be renamed.
dbV4 databases will be moved to a v4 subdirectory.
As of 1/13/20 the Cloud directory will be frozen with no more new entries.
The will be no more updates to dbV4 databases.
The FASTA directory will contain nr, nt, swissprot, and pdbaa files.

If you have any questions or concerns, please contact blast-help@ncbi.nlm.nih.gov

Mycology Research Resources for Bioinformaticians: Unlocking the Fungal Kingdom

Neel — Fri, 13 Dec 2024 11:21:45 -0600

Mycology, the study of fungi, is a field that bridges ecology, medicine, and biotechnology. With advancements in bioinformatics, researchers now have unprecedented opportunities to explore the fungal kingdom at molecular, genetic, and ecological levels. From understanding pathogenic fungi to harnessing fungal enzymes for industrial applications, the potential is vast.

To fully leverage these opportunities, bioinformaticians require specialized tools and databases. This blog highlights essential resources for mycology research, focusing on databases, tools, and platforms tailored for fungal biology.

1. Fungal Databases

1.1. MycoCosm

Website: MycoCosm
Developed by the DOE Joint Genome Institute, MycoCosm is a comprehensive portal for fungal genomics. It offers genomic and transcriptomic data for a wide range of fungi, including saprobes, pathogens, and symbionts.

Key Features: Genome browsers, comparative genomics tools, and functional annotations.
Best For: Large-scale studies on fungal evolution and ecology.

1.2. FungiDB

Website: FungiDB
FungiDB is an integrated genomic resource for fungal pathogens and non-pathogens. It provides access to genome sequences, transcriptomic data, and functional annotations.

Key Features: Advanced search options, BLAST, and pathway analysis tools.
Best For: Studying fungal pathogenesis and host-pathogen interactions.

1.3. Index Fungorum

Website: Index Fungorum
This nomenclatural database provides information on the scientific names of fungi. It’s an essential resource for taxonomists and researchers focused on fungal biodiversity.

Key Features: Taxonomic hierarchy and synonymy tracking.
Best For: Identifying and classifying fungal species.

1.4. UNITE

Website: UNITE
UNITE is a specialized database for fungal ITS (Internal Transcribed Spacer) sequences, often used in fungal identification and phylogenetics.

Key Features: Curated reference datasets and community annotations.
Best For: Environmental mycology and microbial ecology studies.

2. Analytical Tools

2.1. Funannotate

Repository: GitHub - Funannotate
Funannotate is a genome annotation tool designed for fungi. It supports tasks like gene prediction, functional annotation, and orthology analysis.

Best For: Annotating newly sequenced fungal genomes.

2.2. BUSCO (Benchmarking Universal Single-Copy Orthologs)

Website: BUSCO
BUSCO evaluates genome assembly and annotation completeness using orthologs. It includes a fungal-specific dataset.

Best For: Assessing the quality of fungal genome assemblies.

2.3. Pathogen-Host Interactions Database (PHI-base)

Website: PHI-base
PHI-base is a manually curated resource containing information on pathogen-host interactions, including fungal pathogens.

Best For: Exploring virulence factors and host-pathogen relationships.

3. Visualization Platforms

3.1. Cytoscape

Website: Cytoscape
A powerful tool for visualizing molecular interaction networks, Cytoscape can be used to study protein-protein interactions, gene networks, and metabolic pathways in fungi.

Best For: Network biology and functional genomics.

3.2. iTOL (Interactive Tree of Life)

Website: iTOL
iTOL is an interactive tool for visualizing phylogenetic trees.

Best For: Displaying fungal phylogenies and comparing evolutionary relationships.

4. Community Resources

4.1. Mycological Society of America (MSA)

Website: MSA
The MSA promotes fungal research and provides access to resources, conferences, and publications.

Best For: Networking with fungal researchers and accessing recent studies.

4.2. OpenFungi

Website: OpenFungi
OpenFungi is an open-source initiative providing fungal genomic and transcriptomic datasets for research and education.

Best For: Sharing and accessing public fungal datasets.

5. Genomics Workflows

5.1. Galaxy

Website: Galaxy Project
Galaxy offers a web-based platform for reproducible bioinformatics workflows, including tools for fungal genome and transcriptome analysis.

Best For: User-friendly analysis pipelines without requiring coding skills.

5.2. Snakemake

Repository: Snakemake
A flexible pipeline management tool that supports fungal data processing and analysis.

Best For: Custom workflows for large-scale fungal datasets.

Conclusion

Fungal research is a rapidly growing field with vast implications for medicine, agriculture, and industry. For bioinformaticians, the availability of specialized resources—databases, tools, and community platforms—opens doors to innovative discoveries. Whether you are investigating fungal genomics, studying host-pathogen interactions, or exploring fungal biodiversity, the resources outlined above will empower your research journey.

Dive into these resources and help unravel the mysteries of the fungal kingdom!