Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

Address of the bookmark: https://github.com/lmcai/PhyloHerb/

Requirements
- 3-5 years of relevant bioinformatics experience such as public data mining,
processing, analysing and visualising omics data, etc.
- Ph.D., Masters or Bachelors in Bioinformatics, Biotechnology,
Computational Biology, or related field.
- Understanding of molecular biology and biochemistry.
- Comfort and experience with biological research and data.
- Proficient in a programming language used for bioinformatics such as R or
python.
- Excellent communication skills.
- Ability to summarise and simplify complex analyses for a non-technical
audience.
- Strong analytical skills, curiosity and a knack to solve difficult problems.
- Work well in multi-disciplinary teams with people of vastly different
backgrounds.
- Demonstrated success in collaboration and independent work.

More at https://angel.co/elucidata/jobs/460104-senior-bioinformatics-scientist

This blog provides a comprehensive step-by-step guide to detect piRNAs using bioinformatics tools and workflows.

Step 1: Prepare Your Data

1. Obtain RNA Sequencing Data
   Acquire raw small RNA-seq data in FASTQ format. Datasets can be sourced from repositories like NCBI SRA, EMBL-EBI, or specific small RNA sequencing projects.

2. Quality Control (QC)
   Use FastQC to assess the quality of raw reads:

   fastqc reads.fastq

   Evaluate the per-base quality, adapter content, and overrepresented sequences.

3. Trimming and Adapter Removal
   Use tools like Cutadapt or Trim Galore! to remove adapters and low-quality bases:

   cutadapt -a TGGAATTCTCGGGTGCCAAGG -o trimmed_reads.fastq reads.fastq

   Ensure the remaining reads are of high quality for downstream analysis.

Step 2: Map Reads to the Genome

Mapping reads to the reference genome is crucial for identifying piRNA loci.

1. Reference Genome Preparation
   Download the genome assembly of your organism from databases like Ensembl, UCSC Genome Browser, or NCBI.

2. Align Reads
   Use Bowtie or STAR for small RNA alignment:

   bowtie -v 1 -k 1 --best genome_index trimmed_reads.fastq -S aligned_reads.sam
   • -v 1: Allows one mismatch.
   • -k 1: Reports the best alignment.

3. Convert SAM to BAM
   Convert and sort alignments using SAMtools:

   samtools view -Sb aligned_reads.sam | samtools sort -o sorted_reads.bam

Step 3: Identify Small RNAs

piRNAs are characterized by their size (24–32 nt) and strand bias.

1. Extract Reads by Size
   Use tools like BEDtools or custom scripts to filter reads between 24 and 32 nt:

   bedtools bamtofastq -i sorted_reads.bam -fq all_reads.fastq seqkit seq -m 24 -M 32 all_reads.fastq > piRNA_size_reads.fastq

2. Check for Sequence Bias
   piRNAs often have a strong bias for a uridine at the 5’ end (1U bias). Use tools like WebLogo to visualize sequence motifs.

Step 4: Detect Ping-Pong Signature

The ping-pong amplification loop is a hallmark of piRNA biogenesis, characterized by a 10 nt overlap between piRNAs on opposite strands.

1. Generate Overlap Statistics
   Use the piPipes tool or custom scripts to calculate overlap:

   python ping_pong_overlap.py sorted_reads.bam

2. Visualize Overlap Distribution
   Plot the distribution of overlaps to confirm the presence of the 10 nt ping-pong signature.

Step 5: Annotate piRNA Clusters

piRNAs are often generated from genomic clusters.

1. Cluster Identification
   Use tools like proTRAC or PIRANHA to identify piRNA-producing clusters:

   proTRAC.pl -s sorted_reads.bam -g genome.fa -o clusters

2. Annotate Genomic Regions
   Annotate the identified clusters using gene annotation files (GTF/GFF). Tools like BEDtools intersect can help associate piRNA clusters with genes or transposable elements:

   bedtools intersect -a clusters.bed -b genome_annotation.gtf > annotated_clusters.bed

Step 6: Functional Analysis

Functional analysis of piRNAs can uncover their targets and regulatory roles.

1. Predict piRNA Targets
   Use tools like IntaRNA or RNAhybrid to predict interactions between piRNAs and potential target mRNAs:

   RNAhybrid -t target_transcripts.fa -q piRNAs.fa > piRNA_targets.txt

2. Enrichment Analysis
   Perform GO or KEGG enrichment analysis of target genes using tools like g:Profiler or DAVID.

Step 7: Validation and Visualization

1. Validate piRNA Candidates
   Cross-check the identified piRNAs against known piRNA databases, such as piRBase or piRNAdb.

2. Visualize Results

   • Use IGV (Integrative Genomics Viewer) to visualize piRNA alignment and clusters on the genome.
   • Generate heatmaps or circos plots to present piRNA distributions.

Step 8: Share and Publish Findings

1. Archive Data
   Submit sequencing data to public repositories like SRA or GEO with metadata specifying piRNA-related experiments.

2. Publish Results
   Share findings in journals or conferences, emphasizing novel piRNA candidates, target genes, or regulatory mechanisms.

Conclusion

Detecting piRNAs involves a combination of computational and analytical methods to identify these unique small RNAs and their roles in gene regulation and transposable element suppression. By following this step-by-step guide, you can confidently navigate the complexities of piRNA detection and contribute to the growing understanding of their biological significance.

Also, Bioinformatics Industrial Training Program (BIITP) is merged with the HRD Biotechnology Industrial Training Programme (BITP).

While this comes as a surprise for a lot of participants. We believe this is a good attempt to unify and create a national benchmark for talent. And we appreciate this endeavor from Department of biotechnology.

However, such last-minute announcements can create confusion. Thus candidates are advised to go through the complete notification DBT-BET JRF 2019 via the link below.If you have any kind of doubts, you must contact DBT JRF or Biotecnika for any kind of help & assistance.

Attention:-Bioinformatics Programs (BINC and BIITP)

1. Bioinformatics National Certification (BINC) has been merged with DBT-Junior
Research Fellow (BET Exam)

2. Bioinformatics Industrial Training Program (BIITP) is merged with HRDBiotechnology Industrial Training Programme (BITP).

Students of Bioinformatics, who are interested to apply for Fellowship or Industrial
Training may keep track of the advertisement of DBT-JRF (BET Exam) and BITP
of DBT.

 More at http://www.bcil.nic.in/files/Attention_Bioinformatics_Programs_(BINC_and_BIITP).pdf

No previous programming knowledge is assumed. By following this tutorial you will learn the fundamental concepts of programming by using scripting languages: variables, types, arrays, cycles, conditional statements, functions, objects, regular expressions, files reading and manipulation et-cetera.

Address of the bookmark: http://www.cellbiol.com/bioinformatics_web_development/introduction/

Address of the bookmark: https://github.com/reubwn/hgt

More at http://clark.genetics.utah.edu/

Address of the bookmark: https://pgapx.ybzhao.com/

The bpRNA code is written in perl and requires the Graph perl module. Several additional scripts for analysis are included. The source code is available at http://github.com/hendrixlab/bpRNA.

Address of the bookmark: http://github.com/hendrixlab/bpRNA