Fast queue - ZeroQ
High performace in-memory tasks queue for Node.js. Perfect for executing in batches while utilizing resources to the maximum.
High perormance
ZeroQ was built to endure great tasks load, while preserving the order of the tasks. It surpassing other implementations by:
- It's in-memory data structure is faster - Linked list instead of Array (I measured 200X latency with
[].shift
comparing to changing a single pointer of linked list) - Super-high test coverage
- Out of the box support for TypeScript.
- No dependencies.
I currently don't provide other important queue features such as priority queue as I found current popular implementation fast enough with the features these provide.
Speed was our greatest concern and not a full featured queue.
Simple to use
Using Javascript
//Require all ZeroQ module const ZeroQ = ; //Now you can use both `ZeroQ.DataQueue` and `ZeroQ.TasksQueue` //Or Directly load the desired class const DataQueue TasksQueue = ;
Using TypeScript
; //Now you can use both `ZeroQ.DataQueue` and `ZeroQ.TasksQueue`
API
Both Queues supports maxConcurrency
number as first argument.
This enables locking/releasing multiple resources with a single queue.
Tasks queue - TasksQueue(maxConcurrency : number , isSyncMode : boolean )
This class was created to distribute tasks in the order these were pushed. Tasks will be excuted once locked resource will be released.
const TasksQueue = ;//Create a sync queue with no task executing in parallelconst queue = 1 true;//Push some task to queuequeue;//=> task 1queue;queue;//=> task 2
Data queue - DataQueue(maxConcurrency : number , onData : (data:T)=>void )
This class was created to distribute data in the order these were pushed. Data will be pushed once locked resource will be released.
const DataQueue = ;//Create a sync queue with no task executing in parallelconst queue = 1 { console;};//Any data can be pushed to the queue it will be release into your callback as isqueue;//=>just recived { eventName : 'click'}queue;queue; ;
Examples
Making multiple parallel request
Usually when making requests with Noed.JS the best approch is to request I/O operation to execute in parallel and wait for the last one to return.
const rp = ; Promiseall ;
While it is in most cases the best approch when the amount of requests required is much higher, requesting from the OS to preform so many requests at once will probably result in random recurring timeouts.
The best approach is to lock a big number of requests. As long as the number of requests is lower than the maximum concurrency - all requests will execute together. Else it will be executed in bulks.
const rp = ; const requestQueue = 30; const thousendsOfResources = ; //No matter the number in this case only 30 request will be exectured in parllel for let i =0; i<50 ;i++ thousendsOfResources; Promiseall thousendsOfResources
Synchronous writing to an IPC socket
While Node.js doesn't preform any task in parallel, the OS do. Some of the following messages will be lost because the OS can't write two messages in parallel.
const net = ;const client = net;
With tasks queue it's super safe and simple to make sure all your messages would arrive
const net = ;const writeQueue = 1;const client = net;
Asynchronous proccesing thousands of files
const queue = 1000; const glob = ; const readFile = const getFileNames = { }; { return { } } getFileNames