Live online courses on statistics with R based on this book, led by the author, are offered regularly; see this page for more information and dates.

The past decades have transformed the world of statistical data analysis, with new methods, new types of data, and new computational tools. The aim of Modern Statistics with R is to introduce you to key parts of the modern statistical toolkit. It teaches you:

• Data wrangling - importing, formatting, reshaping, merging, and filtering data in R.
• Exploratory data analysis - using visualisations and multivariate techniques to explore datasets.
• Statistical inference - modern methods for testing hypotheses and computing confidence intervals.
• Predictive modelling - regression models and machine learning methods for prediction, classification, and forecasting.
• Simulation - using simulation techniques for sample size computations and evaluations of statistical methods.
• Ethics in statistics - ethical issues and good statistical practice.
• R programming - writing code that is fast, readable, and (hopefully!) free from bugs.

The book includes plenty of examples and more than 200 exercises with worked solutions. The datasets used for the examples and the exercises can be downloaded here.

Address of the bookmark: https://www.modernstatisticswithr.com/

https://bioinformatics-core-shared-training.github.io/cruk-summer-school-2017/Day1/Session4-seqIntro.html

Address of the bookmark: https://bioinformatics-core-shared-training.github.io/cruk-summer-school-2017/Day1/Session4-seqIntro.html

Address of the bookmark: https://bioinfologics.github.io/post/2018/09/17/k-mer-counting-part-i-introduction/

Prerequisites

This is an intermediate lesson and assumes learners have already done some bioinformatics:

• Familiarity with the BASH command shell, including concepts like pipes, variables and loops.
• Knowledge of bioinformatics fundamentals like the FASTQ file format and transcriptome sequencing, in order to understand the example workflow.

No previous knowledge of Snakemake or workflow systems is required.

https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/index.html

Address of the bookmark: https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/aio/index.html

CORE MODULES

Three modules make up the core engine for MCE.

MCE::Core

Provides the Core API for Many-Core Engine. The various MCE options are described here.

MCE::Signal

Temporary directory creation, cleanup, and signal handling.

MCE::Util

Utility functions for Many-Core Engine.

MCE EXTRAS

There are 4 add-on modules for use with MCE.

MCE::Candy

Provides a collection of sugar methods and output iterators for preserving output order.

MCE::Mutex

Provides a simple semaphore implementation supporting threads and processes.

MCE::Queue

Provides a hybrid queuing implementation for MCE supporting normal queues and priority queues from a single module. MCE::Queue exchanges data via the core engine to enable queuing to work for both children (spawned from fork) and threads.

MCE::Relay

Enables workers to receive and pass on information orderly with zero involvement by the manager process while running.

MCE MODELS

The models take Many-Core Engine to a new level for ease of use. Two options (chunk_size and max_workers) are configured automatically as well as spawning and shutdown.

MCE::Loop

Provides a parallel loop utilizing MCE for building creative loops.

MCE::Flow

A parallel flow model for building creative applications. This makes use of user_tasks in MCE. The author has full control when utilizing this model. MCE::Flow is similar to MCE::Loop, but allows for multiple code blocks to run in parallel with a slight change to syntax.

MCE::Grep

Provides a parallel grep implementation similar to the native grep function.

MCE::Map

Provides a parallel map model similar to the native map function.

MCE::Step

Provides a parallel step implementation utilizing MCE::Queue between user tasks. MCE::Step is a spin off from MCE::Flow with a touch of MCE::Stream. This model, introduced in 1.506, allows one to pass data from one sub-task into the next transparently.

MCE::Stream

Provides an efficient parallel implementation for chaining multiple maps and greps together through user_tasks and MCE::Queue. Like with MCE::Flow, MCE::Stream can run multiple code blocks in parallel with a slight change to syntax from MCE::Map and MCE::Grep.

MISCELLANEOUS

Miscellaneous additions included with the distribution.

MCE::Examples

Describes various demonstrations for MCE including a Monte Carlo simulation.

MCE::Subs

Exports functions mapped directly to MCE methods; e.g. mce_wid. The module allows 3 options; :manager, :worker, and :getter.

REQUIREMENTS

Perl 5.8.0 or later. PDL::IO::Storable is required in scripts running PDL.

SOURCE AND FURTHER READING

The source, cookbook, and examples are hosted at GitHub.

• https://github.com/marioroy/mce-perl

• https://github.com/marioroy/mce-cookbook

• https://github.com/marioroy/mce-examples

SEE ALSO

MCE::Shared provides data sharing capabilities for MCE. It includes MCE::Hobo for running code asynchronously.

• MCE::Shared

• MCE::Hobo

Address of the bookmark: https://github.com/marioroy/mce-examples

Many times in bioinformatics we need to deal with huge datasets which  are more than 100GB size. The traditional way to analysis a file is using the while loop

while (FILE){

Do something;

}

This is very slow(since we are using only one processor) and if we have 500 million lines in the dataset it takes more than a day to iterate through the whole dataset. So how do we make best use of all our processors and get the work done quickly?

Here is a very simple and efficient technique with perl which i have been using. I am  more inclined towards using perl fork than perl threads.

One of the oldest way to fork is

my $fork = fork();
if($fork){   
push (@childs,$fork); 
}
elseif($fork==0){
your code here;
exit(0);
}
else{die “Couldnt fork : $!”;}

## wait for the child process to finish
foreach(@childs){
my $tmp=waitid($_,0);
}

what a fork does is it creates a child process and takes the variables and code with it to analyze it separately (detached from the parent process) and thus a separate process is created( which usually runs on a separate processor). Thats it!! One big disadvantage of forking is its very difficult to share variables among the different processes. I will show you how to do it easily but still it has its own drawbacks.

Okie, now if you really do not want to use fork in your code, that’s okie too..There are many useful modules which do it for you very efficiently. One really useful module is Parallel::ForkManager. You can use Parallel::ForkManager to manage the number of forks you want to generate (number of processors you want to use).

Simple usage:
use Parallel::ForkManager;
my $max_processors=8;
my $fork= new Parallel::ForkManager($max_processors);
foreach (@dna) {
$fork->start and next; # do the fork
you code here;
$fork->finish; # do the exit in the child process
}
$pm->wait_all_children;

so you will be generating 8 forks which do the same thing for your each element of array. when one child finishes, Parallel::ForkManager generates a new one and thus you will be using all your processors to analyze the data. Now, if you have generated 8 child processes and want to write the data to one file. You need to lock the file to do this, because you will have problems with the buffering. You can lock the file using flock command.

open (my $QUAL, “myfile.txt”);
flock $QUAL, LOCK_EX or die “cant lock file $!”;
print $QUAL “$output”;
flock $QUAL, LOCK_UN or die “$!”;
close $QUAL;

I would not suggest using flock when dealing with multiple processes because it will decrease the processing efficiency( each child process must wait for the lock to be released by the other child process). Instead, I would suggest each fork writing to a separate file and after the processing just concatenating them.

Putting it all together, If you have 100GB data you can do this

step 1 : split the dataset equally according to number of processors you have. this may take a few hours(about 2-3 hrs for 100GB file)
You can use unix “split” command for this
for example:
my $number_split=int($number_of_entries_in_your_dataset/$max_processors);
my $split_Files=`split -l $number_split “your_file.fasta” “file_name”`;

step2: open you directory comtaining you split files and start Parallel::ForkManager.
For example:
opendir(DIRECTORY, $split_files_directory) or die $!; ### open the directory
my $fork= new Parallel::ForkManager($max_processors);
while (my $file = readdir(DIRECTORY)) { ### read the directory
if($file=~/^\./){next;}
print $file,”\n”;
########## Start fork ##########
my $pid= $super_fork->start and next;
Whatever you want to do with the split file ;
analyze my piece of $file;
######### end fork ###############
$super_fork->finish;
}
$super_fork->wait_all_children;

So basically each processor will be active with its piece of data (split file) and thus you have created 8 processes at one time which run without interfering with the other process. I again will not suggest writing output from each child process to one file(for reasons above). Write output from each fork to a separate file and finally concatenate them. Thats it, you have just increased your program speed by 8 times!! Isnt it easy?

Note:
You may worry about concatenation of the output each child generates, since it does take some time(remember 100GB). I think now you can use a mysql database LOAD DATA LOCAL INFILE command to load all the files into a single table(Should take about 3hrs for 100Gb dataset) and then export the whole table into one file. This should be faster than just concatenating them using “cat” command.(correct me if I am wrong)

Or much simpler way is to use pipes

cat output_dir/* | my_pipe or my_pipe <(file1) final_file;

Thats it guys!! Enjoy programming and please do comment. I am not a computer scientist so forgive me for any mistakes and if any please report them. Thank you.

1st Livestock Functional Genomics Summer School (LFG 2013).

This School was designed for graduate students and early-stage researchers with interest in livestock genomics, who are engaged in projects that require knowledge in the field of computational biology.

Sixty selected participants will spend 13 days receiving theoretical and practical training in genomic data handling from internationally renowned experts.

After the course, the participant should understand the basis and the context of livestock big molecular data, and be able to manipulate high density genotypes, whole genome sequences and transcriptome data.

The Summer School will be held in Araçatuba-SP Brazil, from the 13th to the 21st of September 2013.

All accepted participants will have *expenses fully covered (air ticket, hotel and meals)*, including a free pass to the 5th International Symposium on Animal Functional Genomics http://www.isafg2013.org.br

Applicants will be selected based on their résumés. Application date is due by August 10th. Results will be announced in August 12th.

Please consult website: http://www.sciencesatellite.org.br/sschool