Many-Core Engine (MCE) for Perl example

Jit — Tue, 31 Jan 2017 05:37:50 -0600

MCE spawns a pool of workers and therefore does not fork a new process per each element of data. Instead, MCE follows a bank queuing model. Imagine the line being the data and bank-tellers the parallel workers. MCE enhances that model by adding the ability to chunk the next n elements from the input stream to the next available worker.

CORE MODULES

Three modules make up the core engine for MCE.

MCE::Core: Provides the Core API for Many-Core Engine. The various MCE options are described here.
MCE::Signal: Temporary directory creation, cleanup, and signal handling.
MCE::Util: Utility functions for Many-Core Engine.

MCE EXTRAS

There are 4 add-on modules for use with MCE.

MCE::Candy: Provides a collection of sugar methods and output iterators for preserving output order.
MCE::Mutex: Provides a simple semaphore implementation supporting threads and processes.
MCE::Queue: Provides a hybrid queuing implementation for MCE supporting normal queues and priority queues from a single module. MCE::Queue exchanges data via the core engine to enable queuing to work for both children (spawned from fork) and threads.
MCE::Relay: Enables workers to receive and pass on information orderly with zero involvement by the manager process while running.

MCE MODELS

The models take Many-Core Engine to a new level for ease of use. Two options (chunk_size and max_workers) are configured automatically as well as spawning and shutdown.

MCE::Loop: Provides a parallel loop utilizing MCE for building creative loops.
MCE::Flow: A parallel flow model for building creative applications. This makes use of user_tasks in MCE. The author has full control when utilizing this model. MCE::Flow is similar to MCE::Loop, but allows for multiple code blocks to run in parallel with a slight change to syntax.
MCE::Grep: Provides a parallel grep implementation similar to the native grep function.
MCE::Map: Provides a parallel map model similar to the native map function.
MCE::Step: Provides a parallel step implementation utilizing MCE::Queue between user tasks. MCE::Step is a spin off from MCE::Flow with a touch of MCE::Stream. This model, introduced in 1.506, allows one to pass data from one sub-task into the next transparently.
MCE::Stream: Provides an efficient parallel implementation for chaining multiple maps and greps together through user_tasks and MCE::Queue. Like with MCE::Flow, MCE::Stream can run multiple code blocks in parallel with a slight change to syntax from MCE::Map and MCE::Grep.

MISCELLANEOUS

Miscellaneous additions included with the distribution.

MCE::Examples: Describes various demonstrations for MCE including a Monte Carlo simulation.
MCE::Subs: Exports functions mapped directly to MCE methods; e.g. mce_wid. The module allows 3 options; :manager, :worker, and :getter.

REQUIREMENTS

Perl 5.8.0 or later. PDL::IO::Storable is required in scripts running PDL.

SOURCE AND FURTHER READING

The source, cookbook, and examples are hosted at GitHub.

BOL: Many-Core Engine (MCE) for Perl example

Many-Core Engine (MCE) for Perl example