https://bioinformaticsonline.com/snippets/owner/biostar?offset=10 https://bioinformaticsonline.com/snippets/view/42820/steps-to-install-conda-on-linux Mon, 08 Feb 2021 01:11:44 -0600 https://bioinformaticsonline.com/snippets/view/42820/steps-to-install-conda-on-linux Download the installer: Miniconda installer for Linux. ( https://docs.conda.io/en/latest/miniconda.html#linux-installers ) Anaconda installer for Linux. ( https://www.anaconda.com/products/individual ) $ chmod 777 Anaconda3-2020.11-Linux-x86_64.sh $ ./Anaconda3-2020.11-Linux-x86_64.sh #Follow the yes/no flags and finish till the end #Check the installation $ conda -V A few other frequently used commands # list all the conda environment available conda info --envs # Create new environment named as `envname` conda create --name envname # Remove environment and its dependencies conda remove --name envname --all # Clone an existing environment conda create --name clone_envname --clone envname]]> BioStar https://bioinformaticsonline.com/snippets/view/42640/set-up-wgd-environment-using-conda Sun, 17 Jan 2021 19:24:15 -0600 https://bioinformaticsonline.com/snippets/view/42640/set-up-wgd-environment-using-conda #Wgd cannot be installed directly with bioconda at present, so it is slightly troublesome to install, because it #depends on a lot of software. wgd depends on the following software #BLAST #MCL #MUSCLE/MAFFT/PRANK #PAML #PhyML/FastTree #i-ADHoRe #Creat the conda ENV and install its deps #Follow the steps (base) ➜ wgd git:(master) conda create -n wgd python=3.7 blast mcl muscle mafft prank paml fasttree cmake libpng mpi=1.0=mpich Collecting package metadata (current_repodata.json): done Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.9.0 latest version: 4.9.2 Please update conda by running $ conda update -n base -c defaults conda ## Package Plan ## environment location: /home/anaconda3/envs/wgd added / updated specs: - blast - cmake - fasttree - libpng - mafft - mcl - mpi==1.0=mpich - muscle - paml - prank - python=3.7 The following packages will be downloaded: package | build ---------------------------|----------------- boost-1.74.0 | py37he5a615d_2 335 KB conda-forge boost-cpp-1.74.0 | h9d3c048_1 16.3 MB conda-forge c-ares-1.17.1 | h36c2ea0_0 111 KB conda-forge certifi-2020.12.5 | py37h89c1867_1 143 KB conda-forge cmake-3.19.3 | h4547794_0 14.0 MB conda-forge libblas-3.9.0 | 7_openblas 11 KB conda-forge libcblas-3.9.0 | 7_openblas 11 KB conda-forge libgfortran-ng-9.3.0 | hff62375_18 22 KB conda-forge libgfortran5-9.3.0 | hff62375_18 2.0 MB conda-forge liblapack-3.9.0 | 7_openblas 11 KB conda-forge libnghttp2-1.41.0 | h8cfc5f6_2 774 KB conda-forge numpy-1.19.5 | py37haa41c4c_1 5.3 MB conda-forge python-3.7.9 |hffdb5ce_0_cpython 57.3 MB conda-forge setuptools-49.6.0 | py37h89c1867_3 947 KB conda-forge ------------------------------------------------------------ Total: 97.2 MB The following NEW packages will be INSTALLED: _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-1_gnu blast bioconda/linux-64::blast-2.5.0-hc0b0e79_3 boost conda-forge/linux-64::boost-1.74.0-py37he5a615d_2 boost-cpp conda-forge/linux-64::boost-cpp-1.74.0-h9d3c048_1 bzip2 conda-forge/linux-64::bzip2-1.0.8-h7f98852_4 c-ares conda-forge/linux-64::c-ares-1.17.1-h36c2ea0_0 ca-certificates conda-forge/linux-64::ca-certificates-2020.12.5-ha878542_0 certifi conda-forge/linux-64::certifi-2020.12.5-py37h89c1867_1 cmake conda-forge/linux-64::cmake-3.19.3-h4547794_0 expat conda-forge/linux-64::expat-2.2.9-he1b5a44_2 fasttree bioconda/linux-64::fasttree-2.1.10-h516909a_4 icu conda-forge/linux-64::icu-68.1-h58526e2_0 krb5 conda-forge/linux-64::krb5-1.17.2-h926e7f8_0 ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.35.1-hea4e1c9_1 libblas conda-forge/linux-64::libblas-3.9.0-7_openblas libcblas conda-forge/linux-64::libcblas-3.9.0-7_openblas libcurl conda-forge/linux-64::libcurl-7.71.1-hcdd3856_8 libedit conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2 libev conda-forge/linux-64::libev-4.33-h516909a_1 libffi conda-forge/linux-64::libffi-3.3-h58526e2_2 libgcc-ng conda-forge/linux-64::libgcc-ng-9.3.0-h2828fa1_18 libgfortran-ng conda-forge/linux-64::libgfortran-ng-9.3.0-hff62375_18 libgfortran5 conda-forge/linux-64::libgfortran5-9.3.0-hff62375_18 libgomp conda-forge/linux-64::libgomp-9.3.0-h2828fa1_18 liblapack conda-forge/linux-64::liblapack-3.9.0-7_openblas libnghttp2 conda-forge/linux-64::libnghttp2-1.41.0-h8cfc5f6_2 libopenblas conda-forge/linux-64::libopenblas-0.3.12-pthreads_h4812303_1 libpng conda-forge/linux-64::libpng-1.6.37-h21135ba_2 libssh2 conda-forge/linux-64::libssh2-1.9.0-hab1572f_5 libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-9.3.0-h6de172a_18 libuv conda-forge/linux-64::libuv-1.40.0-h7f98852_0 lz4-c conda-forge/linux-64::lz4-c-1.9.3-h9c3ff4c_0 mafft bioconda/linux-64::mafft-7.475-h516909a_0 mcl bioconda/linux-64::mcl-14.137-pl526h516909a_5 mpi conda-forge/linux-64::mpi-1.0-mpich muscle bioconda/linux-64::muscle-3.8.1551-hc9558a2_5 ncurses conda-forge/linux-64::ncurses-6.2-h58526e2_4 numpy conda-forge/linux-64::numpy-1.19.5-py37haa41c4c_1 openssl conda-forge/linux-64::openssl-1.1.1i-h7f98852_0 paml bioconda/linux-64::paml-4.9-h516909a_5 perl conda-forge/linux-64::perl-5.26.2-h36c2ea0_1008 pip conda-forge/noarch::pip-20.3.3-pyhd8ed1ab_0 prank bioconda/linux-64::prank-v.170427-hc9558a2_3 python conda-forge/linux-64::python-3.7.9-hffdb5ce_0_cpython python_abi conda-forge/linux-64::python_abi-3.7-1_cp37m readline conda-forge/linux-64::readline-8.0-he28a2e2_2 rhash conda-forge/linux-64::rhash-1.4.1-h7f98852_0 setuptools conda-forge/linux-64::setuptools-49.6.0-py37h89c1867_3 sqlite conda-forge/linux-64::sqlite-3.34.0-h74cdb3f_0 tk conda-forge/linux-64::tk-8.6.10-h21135ba_1 wheel conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0 xz conda-forge/linux-64::xz-5.2.5-h516909a_1 zlib conda-forge/linux-64::zlib-1.2.11-h516909a_1010 zstd conda-forge/linux-64::zstd-1.4.8-ha95c52a_1 Proceed ([y]/n)? y Downloading and Extracting Packages libgfortran-ng-9.3.0 | 22 KB | ################################################################################ | 100% libnghttp2-1.41.0 | 774 KB | ################################################################################ | 100% libcblas-3.9.0 | 11 KB | ################################################################################ | 100% liblapack-3.9.0 | 11 KB | ################################################################################ | 100% certifi-2020.12.5 | 143 KB | ################################################################################ | 100% cmake-3.19.3 | 14.0 MB | ################################################################################ | 100% libgfortran5-9.3.0 | 2.0 MB | ################################################################################ | 100% c-ares-1.17.1 | 111 KB | ################################################################################ | 100% numpy-1.19.5 | 5.3 MB | ################################################################################ | 100% libblas-3.9.0 | 11 KB | ################################################################################ | 100% python-3.7.9 | 57.3 MB | ################################################################################ | 100% boost-cpp-1.74.0 | 16.3 MB | ################################################################################ | 100% setuptools-49.6.0 | 947 KB | ################################################################################ | 100% boost-1.74.0 | 335 KB | ################################################################################ | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate wgd # # To deactivate an active environment, use # # $ conda deactivate #----------------------------------------------------- set uo wgd (wgd) ➜ Tools git:(master) ✗ git clone https://github.com/arzwa/wgd.git (wgd) ➜ Tools git:(master) ✗ cd wgd (wgd) ➜ wgd git:(master) ls doc example LICENSE README.md setup.py Singularity wgd wgd_cli.py (wgd) ➜ wgd git:(master) pip install . Processing /home/urbe/Tools/wgd Requirement already satisfied: numpy>=1.16 in /home/urbe/anaconda3/envs/wgd/lib/python3.7/site-packages (from wgd==1.2) (1.19.5) Collecting bokeh==1.4.0 Using cached bokeh-1.4.0.tar.gz (32.4 MB) Collecting fastcluster==1.1.25 Downloading fastcluster-1.1.25-cp37-cp37m-manylinux1_x86_64.whl (154 kB) |████████████████████████████████| 154 kB 5.8 MB/s Collecting joblib==0.11 Using cached joblib-0.11-py2.py3-none-any.whl (176 kB) Collecting pandas==0.24.1 Downloading pandas-0.24.1-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB) |████████████████████████████████| 10.1 MB 29 kB/s Collecting biopython>=1.75 Using cached biopython-1.78-cp37-cp37m-manylinux1_x86_64.whl (2.3 MB) Collecting click>=7.0 Using cached click-7.1.2-py2.py3-none-any.whl (82 kB) Collecting coloredlogs>=10.0 Using cached coloredlogs-15.0-py2.py3-none-any.whl (45 kB) Collecting ete3>=3.1 Using cached ete3-3.1.2.tar.gz (4.7 MB) Collecting humanfriendly>=9.1 Using cached humanfriendly-9.1-py2.py3-none-any.whl (86 kB) Collecting Jinja2>=2.7 Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB) Collecting MarkupSafe>=0.23 Using cached MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (27 kB) Collecting matplotlib>=3.0.2 Downloading matplotlib-3.3.3-cp37-cp37m-manylinux1_x86_64.whl (11.6 MB) |████████████████████████████████| 11.6 MB 29 kB/s Collecting cycler>=0.10 Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB) Collecting kiwisolver>=1.0.1 Downloading kiwisolver-1.3.1-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB) |████████████████████████████████| 1.1 MB 68.0 MB/s Collecting packaging>=16.8 Using cached packaging-20.8-py2.py3-none-any.whl (39 kB) Collecting pillow>=4.0 Downloading Pillow-8.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB) |████████████████████████████████| 2.2 MB 53.5 MB/s Collecting plumbum>=1.6.7 Using cached plumbum-1.6.9-py2.py3-none-any.whl (115 kB) Collecting progressbar2>=3.39 Using cached progressbar2-3.53.1-py2.py3-none-any.whl (25 kB) Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB) Collecting python-dateutil>=2.1 Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) Collecting python-utils>=2.3.0 Downloading python_utils-2.5.1-py2.py3-none-any.whl (13 kB) Collecting pytz>=2011k Downloading pytz-2020.5-py2.py3-none-any.whl (510 kB) |████████████████████████████████| 510 kB 38.1 MB/s Collecting PyYAML>=3.10 Using cached PyYAML-5.3.1.tar.gz (269 kB) Collecting scipy>=1.2 Downloading scipy-1.6.0-cp37-cp37m-manylinux1_x86_64.whl (27.4 MB) |████████████████████████████████| 27.4 MB 2.0 kB/s Collecting seaborn>=0.9.0 Using cached seaborn-0.11.1-py3-none-any.whl (285 kB) Collecting six>=1.5.2 Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) Collecting tornado>=4.3 Downloading tornado-6.1-cp37-cp37m-manylinux2010_x86_64.whl (428 kB) |████████████████████████████████| 428 kB 61.8 MB/s Collecting sklearn Using cached sklearn-0.0.tar.gz (1.1 kB) Collecting scikit-learn Downloading scikit_learn-0.24.0-cp37-cp37m-manylinux2010_x86_64.whl (22.3 MB) |████████████████████████████████| 22.3 MB 6.0 kB/s Collecting threadpoolctl>=2.0.0 Using cached threadpoolctl-2.1.0-py3-none-any.whl (12 kB) Building wheels for collected packages: wgd, bokeh, ete3, PyYAML, sklearn Building wheel for wgd (setup.py) ... done Created wheel for wgd: filename=wgd-1.2-py3-none-any.whl size=75291 sha256=56a7a19b3f4e535897038e8297166a640a7a6a7fdfeb89f5afde4aaa0208ae67 Stored in directory: /tmp/pip-ephem-wheel-cache-29m8tx5i/wheels/54/f3/0c/285aa1e33c52ac42d7996c0170a686e47362e33f012b2b3669 Building wheel for bokeh (setup.py) ... done Created wheel for bokeh: filename=bokeh-1.4.0-py3-none-any.whl size=23689200 sha256=7daa115fd1b052d429abb6c7082cbbb96defc81716da99ce873760cedf0dbe6f Stored in directory: /home/urbe/.cache/pip/wheels/49/8c/d1/6b8e1f57e542671673cb3d2faee1a9eccb36be2c08a3915498 Building wheel for ete3 (setup.py) ... done Created wheel for ete3: filename=ete3-3.1.2-py3-none-any.whl size=2272998 sha256=4f8c179d7e30d22661fff06e2be08125d837dcff31d4eac36dfb206639ce693f Stored in directory: /home/urbe/.cache/pip/wheels/17/fd/e2/6ac384d8c2484789304657dde01b96d7ab83f4f1dd96d266df Building wheel for PyYAML (setup.py) ... done Created wheel for PyYAML: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44620 sha256=89175438317c730e5db0088255ca2f5a9745368487ed0fd0c7dbf644774c45c0 Stored in directory: /home/urbe/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653 Building wheel for sklearn (setup.py) ... done Created wheel for sklearn: filename=sklearn-0.0-py2.py3-none-any.whl size=1316 sha256=4e2eed235501a23fa4cb58a4faef00b3c824d88a5a6d6b7478d45bdfd09bad6c Stored in directory: /home/urbe/.cache/pip/wheels/46/ef/c3/157e41f5ee1372d1be90b09f74f82b10e391eaacca8f22d33e Successfully built wgd bokeh ete3 PyYAML sklearn Installing collected packages: six, threadpoolctl, scipy, pytz, python-dateutil, pyparsing, pillow, MarkupSafe, kiwisolver, joblib, cycler, tornado, scikit-learn, PyYAML, python-utils, pandas, packaging, matplotlib, Jinja2, humanfriendly, sklearn, seaborn, progressbar2, plumbum, fastcluster, ete3, coloredlogs, click, bokeh, biopython, wgd Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 biopython-1.78 bokeh-1.4.0 click-7.1.2 coloredlogs-15.0 cycler-0.10.0 ete3-3.1.2 fastcluster-1.1.25 humanfriendly-9.1 joblib-0.11 kiwisolver-1.3.1 matplotlib-3.3.3 packaging-20.8 pandas-0.24.1 pillow-8.1.0 plumbum-1.6.9 progressbar2-3.53.1 pyparsing-2.4.7 python-dateutil-2.8.1 python-utils-2.5.1 pytz-2020.5 scikit-learn-0.24.0 scipy-1.6.0 seaborn-0.11.1 six-1.15.0 sklearn-0.0 threadpoolctl-2.1.0 tornado-6.1 wgd-1.2 (wgd) ➜ wgd git:(master) wgd zsh: correct 'wgd' to 'gd' [nyae]? n Usage: wgd [OPTIONS] COMMAND [ARGS]... Welcome to the wgd command line interface! _______ \ ___ `'. _ _ .--./) ' |--.\ \ /\ \\ ///.''\\ | | \ ' `\\ //\\ //| | | | | | | ' \`// \'/ \`-' / | | | | \| |/ /("'` | | ' .' ' \ '---. | |___.' /' /'""'.\ /_______.'/ || ||\_______|/ \'. __// `'---' wgd Copyright (C) 2018 Arthur Zwaenepoel This program comes with ABSOLUTELY NO WARRANTY; This is free software, and you are welcome to redistribute it under certain conditions; Contact: arzwa@psb.vib-ugent.be Options: -v, --verbosity [info|debug] Verbosity level, default = info. -l, --logfile TEXT File to write logs to (optional) --version Print version number -h, --help Show this message and exit. Commands: dmd All-vs.-all diamond blastp + MCL clustering. kde Fit a KDE to a Ks distribution. ksd Ks distribution construction. mcl All-vs.-all blastp + MCL clustering. mix Mixture modeling of Ks distributions. pre Check and optionally rename CDS files. syn Co-linearity analyses. viz Plot histograms/densities (interactively). wf1 Standard workflow whole paranome Ks. wf2 Standard workflow one-vs-one ortholog Ks.]]> BioStar https://bioinformaticsonline.com/snippets/view/42134/bash-script-to-handle-multifasta-files Mon, 17 Aug 2020 05:32:14 -0500 https://bioinformaticsonline.com/snippets/view/42134/bash-script-to-handle-multifasta-files #Convert all lowercase residues to uppercase in a FASTA sequence file $ awk 'BEGIN{FS=" "}{if(!/>/){print toupper($0)}else{print $1}}' input.fasta > output.fasta #Rearrange FASTA sequences according to their length $ awk '/^>/ {printf("%s%s\t",(N>0?"\n":""), $0);N++;next;} {printf("%s",$0);} END {printf("\n");}' input.fasta |\ >awk -F '\t' '{printf("%d\t%s\n",length($2),$0;)}' |\ >sort -k1,1n | cut -f 2- |tr "\t" "\n" > output.fasta #Add ‘>’ at the beginning of headers in a FASTA file $ awk '{if ($0 ~/_/) {printf ">";} print $0; }' input.fasta > output.fasta #Match FASTA headers in two different multi-FASTA files $ awk 'NR=FNR{a[$0];next}$0 in a{print $0}' input1.fasta input2.fasta #Merge all FASTA files in a directory into a single FASTA file $ awk'1' *.fa > all.fa]]> BioStar https://bioinformaticsonline.com/snippets/view/42027/bash-script-to-get-intergenic-region-from-genome-files Sat, 08 Aug 2020 20:09:51 -0500 https://bioinformaticsonline.com/snippets/view/42027/bash-script-to-get-intergenic-region-from-genome-files #For the intergenic region, we will require the size of the chromosomes. wget http://xxx.chrom.sizes cat xxx.chrom.sizes | sed 's/^chr//' | sed 's/Cp/Pt/' > tmp mv tmp xxx.chrom.sizes gunzip -c genome_file.gtf.gz | awk 'BEGIN{OFS="\t";} $3=="gene" {print $1,$4-1,$5}' | bedtools sort -g xxx.chrom.sizes | bedtools complement -i stdin -g xxx.chrom.sizes | gzip > my_intergenic.bed.gz]]> BioStar https://bioinformaticsonline.com/snippets/view/42026/bash-script-to-extract-intronic-fragments Sat, 08 Aug 2020 20:07:46 -0500 https://bioinformaticsonline.com/snippets/view/42026/bash-script-to-extract-intronic-fragments #To obtain introns, we simply need the gene and exonic coordinates; #by subtracting the exonic regions from the genic region, we have the intronic region. gunzip -c genome_file.gtf.gz | awk 'BEGIN{OFS="\t";} $3=="gene" {print $1,$4-1,$5}' | bedtools sort | bedtools subtract -a stdin -b my_exon.bed.gz | gzip > my_intron.bed.gz]]> BioStar https://bioinformaticsonline.com/snippets/view/42025/bash-script-to-get-exon-fragments-from-genome-files Sat, 08 Aug 2020 20:05:53 -0500 https://bioinformaticsonline.com/snippets/view/42025/bash-script-to-get-exon-fragments-from-genome-files #Exons are already defined in the GTF file, so we simply need to print lines that are marked exonic. gunzip -c genome_file.gtf.gz | awk 'BEGIN{OFS="\t";} $3=="exon" {print $1,$4-1,$5}' | bedtools sort | bedtools merge -i - | gzip > my_exon.bed.gz]]> BioStar https://bioinformaticsonline.com/snippets/view/41986/perl-one-liner-to-print-only-non-uppercase-letters Tue, 21 Jul 2020 21:25:33 -0500 https://bioinformaticsonline.com/snippets/view/41986/perl-one-liner-to-print-only-non-uppercase-letters #Go through file and only print words that do not have any uppercase letters. perl -ne 'print unless m/[A-Z]/' dna.fa > dnaOnlyLowercase.fa #To lowercase everything perl -pne 'tr/[A-Z]/[a-z]/' dnaUpperCase.fa >dnawithoutuppercase.fa;]]> BioStar https://bioinformaticsonline.com/snippets/view/41967/reformat-the-multifasta-for-sequence-length Mon, 13 Jul 2020 08:43:44 -0500 https://bioinformaticsonline.com/snippets/view/41967/reformat-the-multifasta-for-sequence-length #awk oneliner to reformat the multifasta sequences awk '!/^>/ {printf "%s", $0; n = "\n"} /^>/ {print n $0; n = ""}' file.fasta | fold -w 100]]> BioStar https://bioinformaticsonline.com/snippets/view/41947/onliner-to-split-the-multifasta-to-singlefasta-files Sat, 04 Jul 2020 22:09:29 -0500 https://bioinformaticsonline.com/snippets/view/41947/onliner-to-split-the-multifasta-to-singlefasta-files #Split the multifasta to singlefasta # Multi fasta #Single fasta awk '$0 ~ "^>" { match($1, /^>([^:]+)/, id); filename=id[1]} {print >> filename".fa"}' sequence.fasta]]> BioStar https://bioinformaticsonline.com/snippets/view/41424/uninstall-a-perl-module Fri, 13 Mar 2020 05:41:10 -0500 https://bioinformaticsonline.com/snippets/view/41424/uninstall-a-perl-module # uninstall_perl_module.pl from PerlTricks.com use 5.14.2; use ExtUtils::Installed; use ExtUtils::Packlist; # Exit unless a module name was passed die ("Error: no Module::Name passed as an argument. E.G.\n\t perl $0 Module::Name\n") unless $#ARGV == 0; my $module = shift @ARGV; my $installed_modules = ExtUtils::Installed->new; # iterate through and try to delete every file associated with the module foreach my $file ($installed_modules->files($module)) { print "removing $file\n"; unlink $file or warn "could not remove $file: $!\n"; } # delete the module packfile my $packfile = $installed_modules->packlist($module)->packlist_file; print "removing $packfile\n"; unlink $packfile or warn "could not remove $packfile: $!\n"; # delete the module directories if they are empty foreach my $dir (sort($installed_modules->directory_tree($module))) { print("removing $dir\n"); rmdir $dir or warn "could not remove $dir: $!\n"; } #Run #perl uninstall_perl_module.pl Bio::SCF]]> BioStar