Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

Effective July 10, 2023, NCBI’s Assembly and Genome record pages now redirect to new NCBI Datasets pages. As previously announced, these updates are part of our ongoing effort to modernize and improve your user experience. NCBI Datasets is a new resource that makes it easier to find and download genome data.   

The following pages have been updated:

• The NCBI Assembly record pages now redirect to the new NCBI Datasets Genome record pages that describe assembled genomes and provide links to related NCBI tools such as Genome Data Viewer and BLAST.  
• The NCBI Genome record pages now redirect to the NCBI Datasets Taxonomy record pages that provide a taxonomy-focused portal to genes, genomes, and additional NCBI resources.   

During this transition, you will have the option to return to the legacy Genome and Assembly record pages. We will remove the legacy pages in early 2024.  

Requirements
- 3-5 years of relevant bioinformatics experience such as public data mining,
processing, analysing and visualising omics data, etc.
- Ph.D., Masters or Bachelors in Bioinformatics, Biotechnology,
Computational Biology, or related field.
- Understanding of molecular biology and biochemistry.
- Comfort and experience with biological research and data.
- Proficient in a programming language used for bioinformatics such as R or
python.
- Excellent communication skills.
- Ability to summarise and simplify complex analyses for a non-technical
audience.
- Strong analytical skills, curiosity and a knack to solve difficult problems.
- Work well in multi-disciplinary teams with people of vastly different
backgrounds.
- Demonstrated success in collaboration and independent work.

More at https://angel.co/elucidata/jobs/460104-senior-bioinformatics-scientist

What is Genome Assembly?

Genome assembly involves piecing together short DNA sequence reads generated by sequencing platforms (e.g., Illumina, PacBio, Oxford Nanopore) into longer, contiguous sequences called contigs. This can be performed as:

• De Novo Assembly: Without a reference genome.
• Reference-Guided Assembly: Using a reference genome to guide the assembly process.

Step 1: Preparing Your Data

Before starting the assembly, ensure that your raw sequencing data is high quality.

1. Input Data

   • Short Reads: Illumina sequencing generates short, accurate reads ideal for scaffolding.
   • Long Reads: PacBio and Nanopore sequencing provide long reads for resolving repetitive regions.

2. Quality Control (QC)
   Use tools like FastQC or MultiQC to assess the quality of your reads:

   fastqc reads.fastq multiqc .

   Look for issues like low-quality bases, adapter contamination, or overrepresented sequences.

3. Read Trimming and Filtering
   Trim low-quality bases and adapters using Trimmomatic or Cutadapt:

   trimmomatic PE reads_R1.fastq reads_R2.fastq trimmed_R1.fastq trimmed_R2.fastq \ ILLUMINACLIP:adapters.fa:2:30:10 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:20 MINLEN:36

Step 2: Choosing an Assembly Strategy

Select an assembly strategy based on your data type:

• Short-Read Assemblers:

  • SPAdes: Popular for microbial genomes.
  • Velvet: Fast for smaller genomes.

• Long-Read Assemblers:

  • Canu: Ideal for long-read datasets.
  • Flye: Versatile for small and large genomes.

• Hybrid Assemblers:

  • MaSuRCA: Combines short and long reads.
  • Unicycler: Optimized for bacterial genomes.

Step 3: Running the Assembly

3.1. SPAdes (Short-Read Assembly)

SPAdes is an excellent choice for small genomes, such as bacteria.

The output includes assembled contigs (contigs.fasta) and scaffolds (scaffolds.fasta).

3.2. Canu (Long-Read Assembly)

Canu is designed for high-error long reads from PacBio or Nanopore.

The output will be in canu_output/genome.contigs.fasta.

3.3. Hybrid Assembly with Unicycler

Unicycler combines short and long reads for improved assemblies.

Step 4: Assessing Assembly Quality

After assembly, evaluate its quality using the following tools:

1. QUAST
   QUAST generates assembly statistics, such as N50, genome size, and GC content:

   quast contigs.fasta -o quast_output

2. BUSCO
   BUSCO checks genome completeness by identifying conserved genes:

   busco -i contigs.fasta -o busco_output -l fungi_odb10 -m genome

3. Assembly Graph Visualization
   Visualize assembly graphs with Bandage:

   Bandage load assembly_graph.gfa

Step 5: Post-Assembly Steps

1. Polishing
   Improve assembly accuracy using tools like Pilon (for short reads) or Racon (for long reads).

   racon long_reads.fasta mapped_reads.sam contigs.fasta > polished_contigs.fasta

2. Scaffolding
   Link contigs into scaffolds using tools like SSPACE or Opera-LG if required.

3. Annotation
   Annotate the assembled genome using Prokka for prokaryotes or Maker for eukaryotes.

   prokka --outdir annotation_output --prefix genome contigs.fasta

Step 6: Sharing and Archiving

1. Submit to Public Repositories
   Share your assembly in databases like NCBI GenBank, ENA, or DDBJ.

2. Metadata Preparation
   Include detailed metadata for your submission, such as organism name, sequencing platform, and coverage.

Best Practices

• Always perform quality checks at each stage to ensure data integrity.
• Use multiple tools to cross-validate results when working with complex genomes.
• Document parameters and software versions for reproducibility.

Conclusion

Genome assembly is a powerful process that transforms raw sequencing data into a coherent representation of an organism’s genome. By following this step-by-step guide, you can successfully assemble genomes and uncover valuable biological insights. Whether you’re assembling a microbial genome or tackling the complexities of a eukaryotic genome, these tools and strategies will set you on the path to success.

Also, Bioinformatics Industrial Training Program (BIITP) is merged with the HRD Biotechnology Industrial Training Programme (BITP).

While this comes as a surprise for a lot of participants. We believe this is a good attempt to unify and create a national benchmark for talent. And we appreciate this endeavor from Department of biotechnology.

However, such last-minute announcements can create confusion. Thus candidates are advised to go through the complete notification DBT-BET JRF 2019 via the link below.If you have any kind of doubts, you must contact DBT JRF or Biotecnika for any kind of help & assistance.

Attention:-Bioinformatics Programs (BINC and BIITP)

1. Bioinformatics National Certification (BINC) has been merged with DBT-Junior
Research Fellow (BET Exam)

2. Bioinformatics Industrial Training Program (BIITP) is merged with HRDBiotechnology Industrial Training Programme (BITP).

Students of Bioinformatics, who are interested to apply for Fellowship or Industrial
Training may keep track of the advertisement of DBT-JRF (BET Exam) and BITP
of DBT.

 More at http://www.bcil.nic.in/files/Attention_Bioinformatics_Programs_(BINC_and_BIITP).pdf

No previous programming knowledge is assumed. By following this tutorial you will learn the fundamental concepts of programming by using scripting languages: variables, types, arrays, cycles, conditional statements, functions, objects, regular expressions, files reading and manipulation et-cetera.

Address of the bookmark: http://www.cellbiol.com/bioinformatics_web_development/introduction/

Applications are invited for one position of Junior Research Fellow (JRF) in a Department of Biotechnology (DBT) sponsored research project entitled “Design, synthesis and evaluation of potent aminopeptidase inhibitors for malarial therapy” under the supervision of Dr. Shakti Sahi.

The monthly fellowship of JRF will be Rs 12,000/- plus HRA as per the University rules.

Essential Qualification: Master degree in any discipline of Life Science with NET qualified.

Desirable Qualification: Preference will be given to candidates having research experience in in silico drug designing/Bioinformatics.

Interested candidates may send their resume to undersigned on or before 14th July 2013 by post-mail/e-mail shaktis@gbu.ac.in or shaktisahi@gmail.com. No TA and DA will be paid for appearing for the interview. Dr. Shakti Sahi (Principle Investigator)

Advertisement:
www.gbu.ac.in/Recruitment/JRF_Advt_DBTProject_Shakt

More at http://clark.genetics.utah.edu/

Our long-term research objective is to understand microtubule organization in living cells, with an emphasis on mitosis. We develop in-vitro assays, quantitative image analysis and cytosim, a computer simulation to study cellular architecture from a mechanistic angle, modeling the interactions of microtubules and related proteins such as molecular motors. In the past, we combined simulations and experiments to study microtubule self-organization, and the mechanical stability of two interacting asters. More recently, we looked at the focusing of mitotic fibers, the formation of antiparallel arrays of microtubules in fission yeast and the spindle positionning in C. elegans.
We are supported by BioMS, an initiative in Systems Biology, and involved in Cell networks.

Link: http://www.cytosim.org