concu
A simple method wrapper for concurrent processing of an array in javascript.
Introduction
The Concu library helps you processing series of data (typically an array) by chunking your data in smaller bits and concurrently processing these chunks. This is typically useful if you have the possibility of processing data in parallel, however you want to have control over:
- how many concurrent executions
- size of the batch being processed
Limiting concurrency and batch size may be needed because of downstream I/O dependencies.
When opted for concurrency of 1, the method presents a simple sequential execution on partial data chunks.
The original focus was on helping to process large CSV files for which each row had a lot of IO, and each row could be processed independently of any other row.
Internals
Concu is basically a simple wrapper around around pLimit and Promise.all()
Usage
Typescript example
// create an array with 10000 elementsconst array = Array1000;const dataToProcess: number = ...array; // Dummy method to just calculate the sumconst processData = numbers: number: { return numbers;} const concurrency = 10; // number of parallel executionsconst chunkSize = 100; // size of every chunk const run = async : Promise<number> return await ; ;
API
concu(fun, chunkSize, concurrency, data, ...args)
returns a Promise resolving to an array of individual results of each individual execution
fun: (data: any[], ...args: any[]) => any
function which accepts a data array and other possible arguments.
chunkSize: number
How many elements of the array to be processed
concurrency: number
how many executions in parallel
...args: any[]
optional arguments that will be passed to fun
Acknowledgments
This work has been created at Locai Solutions as part of the data import efforts.