Address of the bookmark: https://sourceforge.net/projects/contiguator/

It takes FASTA DNA sequence as input, and write GFF3 as output. It uses the new NHMMER tool that comes with HMMER 3.1 for HMM searching in RNA:DNA style. NHMMER binaries for 64-bit Linux and Mac OS X are included and will be auto-detected. Multithreading is supported and one can expect roughly linear speed-ups with more CPUs. 

Address of the bookmark: https://github.com/tseemann/barrnap

Dr Altan Kara is a Bioinformatics specialist at the faculty of Gene Engineering and Biotechnology Institute at TUBITAK MAM Research Center. His research interest revolves around the cancer informatics and computational aided-drug design. I applaud Dr Altan for clearly setting out both his expectations of people that join his lab/university in addition to listing his responsibilities to his research members at TUBITAK MAM Research Institüte. Hopefully, this interview will prove useful to others in the field, especially to those who are just starting their bioinformatics careers.

You can find out more about Dr Altan by visiting his (well documented) lab page (http://gmbe.mam.tubitak.gov.tr/en) and BOL page http://bioinformaticsonline.com/profile/altan . And now, on to the BOL:“Tryst with a Bioinformatician” interview series ...

• What push you to join Computational Biology/Bioinformatics?

According to me, bioinformatics is the center of modern biological research and if a researcher wants to discover new biological insights by evaluating the globally produced biological data to derivate unified solutions for specific biological problems, learning bioinformatics is the only way to achieve this goal.

• What fascinates you about Computational Biology/Bioinformatics?

It's flexibility. As well known, there are highly diverse and complex biological questions are waiting to be enlightened and it's impossible to bring solutions to this diversity by using similar approaches. Thus, the employed method has to be unique for the targeted biological problem and by using bioinformatics tools this can be easily achieved. 

• What is the one word you would use to describe yourself?

Bioinformatician. :)

• Can you please describe your research work in a nutshell for BOL users.

At my current Institute, I am working in the field of cancer bioinformatics. Briefly, the overall aim of the project which I am working for (AKMARK (Project CODE:5153403)) is, applying a bioinformatics-supported genome, transcriptome, proteome, and metabolome analysis to reveal the molecular profile of the disease through an integrated approach, and to develop an early diagnosis and scanning kit based on this profile. Alterations in the gene, transcript, protein, and metabolite profiles between normal tissue, normal tissue adjoined to the tumor (reactive stroma), tumor tissue, lymph node metastasis, and blood samples taken from the same patient and the reflection of these changes in some other selected body fluids will be revealed within the scope of the project. The molecular structures involved in the development and progression of NSCLC will be determined and relations with the clinical, tumor-node-metastasis (TNM) staging and histology will be made. The development of a diagnostic kit for immediate clinical purposes and an electrochemical biosensor for quick on-site applications are targeted through the development of a number of antibody and aptamer formed against the most specific biomarker selected from the panel.

• Is there anything else we should know about you and your research?

Besides AKMARK, I am also in preparation of having a side project that aims for the development of a computational method to design inhibitors for prokaryotic two-component systems. In this project, I will be in collaboration with Prof. Maria Kontoyianni, SIUE: Southern Illinois University Edwardsville, School of Pharmacy.

• What was your greatest scientific disappointment in life till now?

So far I do not experience any memorable scientific disappointment in my life. :)

• What major research challenges and problems did you face yet? How did you handle them?

The major challenge which I faced so far in my scientific career was predicting the interaction between the prokaryotic two-component proteins. To be able to accurately predict the interactions between these proteins, I create a meta-predictor by using a support vector machine. By using this technique I integrated six different protein-protein interaction methods in a way to cover disadvantage of one method with the advantage of another one. The meta-predictor which I developed during this work is accessible via http://metapred2cs.ibers.aber.ac.uk/ and for more detailed information about the system the articles with the PMID IDs; PMID: 27378293 and PMID: 26384938 can be read.

• What's your all-time favourite bioinformatics package, and why?

For me, the best bioinformatics package is R/Bioconductor. The reason why I like this package is, it provides lots of useful tools for comprehensive analysis and comparison of high-throughput experimental data in an integrated manner and besides lots of the packages it provides, it is open source and also open for development. As a result, it provides strong and flexible ways to do science.

• In bioinformatics, do you see yourself in which of the following roles-scientist, analyst, developer, engineer or pure academician?

Scientist / Developer.

• What will you like to accomplish in next five years / ten years?

For my current research, I would like to design a pipeline to automatically integrate and analyse omics data for cancer research which will be specifically aiming for biomarker and novel drug target discovery. In addition to this, I also like to develop another pipeline for prokaryotic TCS protein structure prediction and inhibitor design.

• When you will be retired, what would you tell next generation bioinformaticians?

Bioinformatics is not all about scripting and researchers who study in this field should never expect a tool to do their analyses for them. Besides computational skills, a bioinformatician must have a strong biological background in his/her research area which will allow them to understand if anything went wrong during their run by only looking at the results instead of just blindly trusting the output of the bioinformatics tools.

• What you always miss in bioinformatics when you will no longer working in this field?

Bioinformatics is open to doing multi-discipliner research with scientists all around the world. As a result, while I studying in this field I can interactively learn a lot from wide range research community. I think this is the one thing which I will miss the most.

• If there will be bioinformatics company owned by you in future, What are your company focus and aim?

With the increasing amount of data in databases, there is already a massive need for effective methods to eliminate the manipulated data and reach to clean/useful information. As days pass, the requirement of data mining will be the first step of any research project. For this reason, the major goal of my bioinformatics company will be developing effective tools to eliminate manipulated datasets and information that exist in the literature and provide trustworthy clean information/datasets for researchers.

• How much bioinformatics change in 2050, according to your wild imagination?

Bioinformatics is a field that constantly and dynamically changes. As the bioinformatics progress, new tools and methods become available and they provide a better application of existing methods or totally new methods that offer an alternative solution to various biological problems. A long with these updates, developers also provide easy to use GUIs for most of the tools. Considering this, if the field carries on developing like this, every single researcher with a strong biological background can be able to perform bioinformatics analyses by him/herself without needing a professional help. As a result, almost all of the bioinformaticians will be responsible just for development of new methods/tools.

• What would one piece of advice you give someone who's trying to reinvent themselves and enter into bioinformatics sector?

Bioinformatics is a wide field with a lot of career options. Thus, if a researcher likes to step into this field first he/she should be clear about the branch of the bioinformatics they like to study in. Following to this decision they should first learn at least one programing language and investigate the ways of how other researcher employed that language in their researches and WHY? A researcher, in this field, should never create and use copy paste scripts but always must understand WHY the other researcher worked in that way. Knowing the answer of this question is the only way to learn bioinformatics. Besides, a researcher in the field of bioinformatics (from any branch) must always be good about the environmental control. In other words, one should always easily control input output directories, modify files or directories, annotate and modify employed scripts during the research and should not allow any confusion during the different stages of the research. Finally, they should not blindly trust the output of a tool/software but do a benchmarking test for each of the tools which they decided to utilise in their research. In addition to this, even if the tools pass the benchmarking, researchers should have a good biological background in their field to tell if anything when wrong during the process by only looking the output(s) of the employed pipelines/packages/tools.  

http://software.broadinstitute.org/viral/docs/vicuna_v1.0.pdf

Address of the bookmark: https://www.broadinstitute.org/viral-genomics/vicuna

Perl's second wave of adoption came from the growth of the world wide web. Dynamic web pages—the precursor to modern web applications—were easy to create with Perl and CGI. Thanks to Perl's ubiquity as a language for system administrators and its power to manipulate text, it was the default choice for web programming. Its presence everywhere made it popular and, in some ways, the duct tape of the Internet.

Web Application Development

The old days of CGI programs and the simple development style that represented seem clunky. Web pages have become web applications. Development has moved from generating static HTML to both client and server side programming, with rich client interfaces and powerful backends.

Perl is still well suited for developing modern web apps. The language grows more powerful and easier to use every year, the available libraries are wonderful and keep getting better, and the inventions and discoveries available in modern Perl are unsurpassed.

In particular, a modern Perl developer can do amazing things with modern Perl tools. If you still think of Perl web development as a cgi-bin directory full of messy scripts that spew warnings to STDERR, you're a decade out of date. Better yet, you can replace that mess piecemeal, thanks to the new tools and techniques of modern Perl. See, for example, the ever-growing list of technologies Built in Perl.

Modern Perl Web Frameworks

While the old wave of web development may have made the CGI.pm module central, modern Perl web programming follows a stricter separation of business logic, URL and request routing, and output. The days of slinging a string here, an array there, a Perl hash yonder, declaring every variable at the top of the program, and maybe making a subroutine are gone. The Perl world has seen the value of abstraction and ways to mechanize away boilerplate. Perl has dozens of frameworks and toolkits designed to make web development and deployment simpler.

Any of a dozen of these frameworks will help you do great things, but three in particular stand out. You can build web sites and web applications of tremendous value with all three. These are neither the only good possibilities (think of POE or Jifty or Continuity or...) nor the only mechanisms for web programming with Perl (see Mechanize or LWP or Mojo::UserAgent for more). Yet if you want three good options to choose between, start here.

Catalyst

The Catalyst framework is a flexible and powerful system for building small to large web apps. It uses the Moose object system to provide great APIs for extension and further development. It's the most mature of the modern top Perl web frameworks, yet it retains its flexibility and vibrancy. In particular, its plugin and extension ecosystem allows it to evolve to provide new and essential features.

Catalyst has embraced the Plack/PSGI standard for Perl web deployment and recent versions are exploring high-scalability, event-based request handling models.

Dancer

The Dancer framework is deliberately minimal in syntax and scope, but it also has a vibrant plugin ecosystem. Dancer particularly excels for smaller sites and applications, though good programmers can build larger things with it.

The first version of Dancer was easy to use. Dancer 2 continues that ease while improving the internals and robustness of applications.

Mojolicious

The Mojolicious (Mojo) framework has a real-time design based on high performance event handling. Its focus is solving new and interesting problems in simple and effective ways, and the project has produced a lot of new code that does old things in better ways.

In particular, Mojolicious goes to great lengths to support new web standards, such as CSS 3, web sockets, and HTTP 2.

Where Catalyst embraces the CPAN fully, Mojolicious by design provides most of what an average app might need in a single download. It's still fully compatible with the CPAN, but the intention is to provide good working defaults in a package that's easy to start with. Mojo's fans are quick to praise it as fun to develop.

A modern Perl web developer should be familiar with at least one of these frameworks.

Modern Perl Storage Mechanisms

Perl's venerable DBI module has been the focal point of database access since its invention. Its design allows it to provide the same interface to huge relational databases and flat files alike through its DBD extension mechanism. Yet the DBI by itself isn't the be-all, end-all of data storage and access in Perl.

DBIx::Class

DBIx::Class sits on top of DBI to provide an API to your database based on the concept of queries and results. This is often sufficient to remove all but the most complicated of SQL from your code, leaving you to manipulate your business models instead of the small details of how a relational database works. The power and maintainability you receive is well the small cost of the learning curve.

Even better, DBIC can manage (and even generate) your database schema for you.

Recent versions of DBIC have demonstrated that a well-written ORM can perform much better than even clever hand-written code. Because it builds on the Perl DBI, it scales everywhere from SQLite to PostgreSQL, MySQL, Oracle, and more.

Rose::DB

The lesser-known but no less powerful Rose::DB::Object builds on Rose::DB to provide an object-relational mapper for Perl. While its high level features most directly compare to those of DBIx::Class, it's often measurably faster.

NoSQL on the CPAN

Of course the CPAN has modules for almost any NoSQL database or job queue or persistence mechanism you could name, and several you have never heard of. Everything you need is a quick CPAN or cpanm away!

Modern Perl Deployment Strategies

In the early days of the web, deploying a Perl web application meant putting one or more .cgi or .pl files in a special directory and hoping that your system administrator had everything configured correctly. The execution model was often slow and cumbersome, and accessing shared resources such as databases was often tricky.

Modern Perl has better choices. While deployment strategies are the source of many arguments, the return on your investment from learning the modern way is impressive.

Plack/PSGI

The PSGI specification (as exemplified by Plack) describes a strategy for building Perl web apps independent of server and with the possibility to share custom processing behaviors.

In other words, it's a standard for writing Perl apps to take advantage of the huge ecosystem of Perl development available on the CPAN without tying yourself to a server like Apache, Apache 2, nginx, or anything else.

Any good modern Perl web framework (including those listed here) supports PSGI. Several deployment mechanisms exist to meet various business needs which also support PSGI. In particular, you can deploy the same application with a local testing server on your own machine as you can to your production server or servers without changing your application at all.

mod_perl

The older but still viable mod_perl Apache httpd module embeds Perl into the web server. This was the first widespread persistence mechanism for Perl web applications themselves and it's still popular to this day, though PSGI compliance is often the choice for new development. (PSGI handlers to use mod_perl as the backend are available.)

Modern Perl developers should familiarize themselves with PSGI and the wealth of available Plack middleware.

Perl Web Development

Of course no discussion of Perl web development would be complete without mentioning the strength of the CPAN. Almost any project will benefit from the wealth of freely available libraries built to solve real problems. These distributions run the gamut from full-blown web frameworks and content management systems to APIs for web services, development tools, testing systems, and interfaces to document formats and external resources.

For example, if you need to write a web service which accepts JSON data and produces Excel spreadsheets, you can glue together a few CPAN distributions and get the job done early. If you need to consume XML from a remote service and emit a PDF, you're in luck.

Perl's prowess as a general purpose programming language as well as its flexibility and power in managing text and gluing systems together make it a wonderful fit for web development. The community's adoption of modern Perl standards such as PSGI and Plack only enhance your power.

Web application development in Perl is still viable, and modern Perl tools and techniques and libraries make it more powerful and pleasant than ever.

https://www.biorxiv.org/content/biorxiv/early/2021/02/01/2021.01.29.428779.full.pdf

Address of the bookmark: https://github.com/nadegeguiglielmoni/GraphUnzip

https://academic.oup.com/ve/article/7/1/veab042/6248116

Address of the bookmark: https://academic.oup.com/ve/article/7/1/veab042/6248116

sed ':a;N;$!ba;s/\n//g' filename.txt > newfilename_oneline.txt

To get average for a column of numbers (here the second column $2):

awk '{ sum += $2; n++ } END { if (n > 0) print sum / n; }'

To get sequence length for all sequences in a fasta file:

awk '/^>/ {if (seqlen){print seqlen}; print ;seqlen=0;next; } { seqlen = seqlen +length($0)}END{print seqlen}' \
filename.fasta

To copy (move, rename, etc) files based on their list in a text file:

cat file_list.txt | while read line; do cp "$line" complete_dataset/"$line"; done

To split bam files into sets with mapped and unmapped reads:

samtools view -F4 sample.bam > sample.mapped.sam
samtools view -f4 sample.bam > sample.unmapped.sam

To gzip all your fastq files using gnu parallel and gzip:

parallel gzip ::: *.fastq

To gzip all your fastq files using pigz:

pigz *.fastq

To count all sequences in a fasta file:

grep "^>" yourfile.fasta -c

To count all sequences in all fasta files in your current directory:

for a in *.fasta; do ls $a; grep "^>" -c $a; done

To keep only one copy of duplicated lines:

awk '!seen[$0]++'

To sum assembly size from SPAdes contigs.fasta or scaffolds.fasta file:

grep "^>" scaffolds.fasta | cut -f 4 -d '_' | paste -sd+ | bc

To remove everything after the first space at each line, e.g. to to simplify fasta headers:

cut -d' ' -f1 < your_file

To count reads in a all .fastq.gz files in your current folder (fast, using gnu parallel):

parallel "echo {} && gunzip -c {} | wc -l | awk '{d=\$1; print d/4;}'" ::: *.gz

To count reads in a all .fastq.gz files in your current folder:

zcat *.gz | echo $((`wc -l`/4))

To count reads in a all .fastq files in your current folder:

cat *.fastq | echo $((`wc -l`/4))

To count base pairs in a all .fastq.gz files in your current folder:

zcat *.fastq.gz | paste - - - - | cut -f 2 | tr -d '\n' | wc -c 

To split multifasta file into many fasta files:

awk '/^>/ {OUT=substr($0,2) ".fa"}; {print >> OUT; close(OUT)}' Input_File

To convert Illumina FASTQ 1.3 to 1.8:

sed -e '4~4y/@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghi/!"#$%&'\''()*+,-.\/0123456789:;<=>?@ABCDEFGHIJ/' f.fastq

To convert FASTQ to FASTA:

sed -n '1~4s/^@/>/p;2~4p' 

To get fastq read length distribution:

cat reads.fastq | awk '{if(NR%4==2) print length($1)}' | sort | uniq -c

To deinterleave interleaved fastq file:

cat myf.fq | paste - - - - - - - - | tee >(cut -f 1-4 | tr "\t" "\n" > myfile_1.fq) | cut -f 5-8 | \
tr "\t" "\n" > myf2.fq 

To filter and sort contig identifiers from SPAdes assembly (e.g. here lenght >= 4000 + coverage >=100):

grep "^>" scaffolds.fasta | sed s"/_/ /"g | awk '{ if ($4 >= 4000 && $6 >= 100) print $0 }' | sort -k 4 -n | \
sed s"/ /_/"g

To append something to all headers of your fasta files:

sed 's/>.*/&YOURSTRING/' filename.fasta > new_filename.fasta

To replace/squeeze multiple adjacent spaces by only one space: 

tr -s " " < file

To filter fastq based on length (here larger than or equal to 21, but smaller than or equal to 25.

cat your.fastq | paste - - - - | awk 'length($2)  >= 21 && length($2) <= 25' | sed 's/\t/\n/g' > filtered.fastq

To print difference between the last and first row in 5th column:

awk '{if (!first){first=$5;}; last=$5;} END {print last-first}' myfile.txt

To sample only 200 first bases from all sequences in a multifasta file (e.g. from assembly scaffolds.fasta file here):

awk '/^>/{ seqlen=0; print; next; } seqlen < 200 { if (seqlen + length($0) > 200) $0 = substr($0, 1, 200-seqlen);\
 seqlen += length($0); print }' scaffolds.fasta > 200bp_scaffolds.fasta

 To pipe a compressed fasta file directly into makeblastdb.

gunzip -c fasta.gz | makeblastdb -in -

To remove sequences with duplicate fasta headers from a fasta file.

awk '/^>/{f=!d[$1];d[$1]=1}f' in.fasta > out.fasta