pelias-parallel-stream
Sometimes you want to call an async function inside a transform stream and be able to take full advantage of the magical asynchronicity of node.js.
In order to do this, we must decouple the this.push(data)
from the next()
.
This module allows you to do that in a clean and simple way. See below.
usage
var parallelStream = ; var maxInFlight = 2; var seeYouLaterStream = ;
NOTE: the end function is optional
Once you've made your new parallel stream, you can use it just like you would any normal transform stream.
Just throw it into a .pipe()
call, like so.
var streamArray = ;var sink = ; // <--- there it is ;
You can see the full example here
Now let's play around with this maxInFlight
parameter.
Let's first see how setting the maxInFlight
to 1
results in a strandard serial transform stream.
So the output will look like this...
$ time npm run example -- 1I see you, DianaOh hey there again, DianaI see you, JulianOh hey there again, JulianI see you, StephenOh hey there again, StephenI see you, PeterOh hey there again, Peter real 0m4.256suser 0m0.114ssys 0m0.021s
Now let's set it to 2
and see how different the output looks, and if performance has improved.
$ time npm run example -- 2I see you, DianaI see you, JulianOh hey there again, DianaI see you, StephenOh hey there again, JulianI see you, PeterOh hey there again, StephenOh hey there again, Peter real 0m2.258suser 0m0.128ssys 0m0.025s
You can see when we allow 2 requests in flight, we get the first 2 requests back-to-back, send then off for async
handling and then pause to wait for one of them to return and make room for the next incoming request.
As soon as we've seen one of the first 2 requests come back (Oh hey there again, Diana
),
another incoming requests comes in (I see you, Stephen
). And let's note that the amount of time it took to get through
all the data has been cut in half, because... asynchronous!
Finally, let's run it with maxInFlight
set to 200
, which is just a number larger than the length of the input data array.
$ time npm run example -- 200I see you, DianaI see you, JulianI see you, StephenI see you, PeterOh hey there again, DianaOh hey there again, JulianOh hey there again, StephenOh hey there again, Peter real 0m1.159suser 0m0.121ssys 0m0.022s
You can see that all the requests were sent out at once, and all the responses came in shortly thereafter. Note how quickly it all happened, too.
Versioning
We rely on semantic-release and Greenkeeper to maintain our module and dependency versions.