supertask-cluster
Create, compile and run tasks across a cluster with minimal setup.
Installation
Note that this module is a superset of Supertask and requires ES6 & NodeJS 4.x+.
npm install supertask-cluster
Features
- Automatic Cluster & Worker creation
- Cluster Monitoring, Worker resurrection
- Bidirectional argument passing and results from Master to Worker and vice versa.
- Processing & Compilation of function or JS source on Workers with minimal effort. (uses NodeJS VM no interpreters or parsers)
- Buffer upload mechanism and Buffer references as arguments
- Entirely written in JS and fully based on NodeJS core modules where possible
Usage
Create a new shared task with a unique name and distribute it to all available Workers (one per core). Note that a unique name is required for every task.
var SupertaskCluster = ;var cluster = ;// Deploy Cluster across all CPU corescluster; TaskManager;
Run task. You can pass arguments after name and before callback. This will automatically run the task on a free Worker based on the internal load.
TaskManager;
Note that the usual Supertask methods work as this module is merely a superset.
API
Buffers
Buffers allow you to pre-upload a StorageType (currently only NodeJS's internal Buffer object is supported) to a Worker in order to save upload time. This is inspired by GPU's parallelization mechanism whereby data is uploaded to the GPU in order to save bandwidth and data transfer time that will otherwise make parallelization on GPUs slow. This same concept applies to a Cluster and if large Buffers are passed as arguments to every function every time you'll spend most of your processing time uploading data. This is where Buffers come in. Buffers are automatically chunked & uploaded (like a stream) to avoid congestion that would otherwise make passing of a 2GB Buffer to a Worker impossible.
Here is how to use Buffers:
Create a new task:
// Create a large 20MB buffer on Worker ID 0var buffer = 20000000;// Fill buffer with 'c'sbuffer;// Create new Buffer on Worker 0. Note that the Buffer will take some time to upload.// (approximately 20s per 1GB or 400ms for 20MB)TaskManager;
Note that the Cluster automatically chooses the best candidate that has the Buffer reference available. If the Buffer reference is not available in any of the workers it will be set to undefined
.
A Buffer can be immutable. Read API documentation for more info. More documentation and methods coming soon. Check out the test functions for more information in the mean time.
Disclaimer
This module is not yet ready to be used in a production environment. While Supertask has reasonably good stability with over 40 tests it does not fully expose all methods and capabilities and may not function as intended. Supertask-cluster is equally missing some important cluster monitoring methods to keep the cluster alive and well in a production environment. Use it at your own risk.
License
MIT © Schahriar SaffarShargh info@schahriar.com