Async waterfall in node.js

Recently, I had to use aws lambda which moves a file from one s3 bucket to another. So like any developer would do, I looked into some documentation and stumbled upon async-waterfall implementation in their documentation.

What can you do with async-waterfall?

async-waterfall essentially allows you to run a series of functions who’s results can be used as inputs for the following function.

According to this stackoverflow answer,

1. The control flow is specified by an array of functions for invocation as the first argument, and a “complete” callback when the flow is finished as the second argument.
2. The array of functions are invoked in series (as opposed to parallel).
3. If an error (usually named err) is encountered at any operation in the flow array, it will short-circuit and immediately invoke the "complete"/"finish"/"done" callback.
4. Arguments from the previously executed function are applied to the next function in the control flow, in order, and an “intermediate” callback is supplied as the last argument. Note: The first function only has this “intermediate” callback, and the “complete” callback will have the arguments of the last invoked function in the control flow (with consideration to any errors) but with an err argument prepended instead of an "intermediate" callback that is appended.
5. The callbacks for each individual operation should be invoked when you're ready to move on: The first parameter will be an error, if any, and the second (third, fourth... etc.) parameter will be any data you want to pass to the subsequent operation.

Picture courtesy : Penta-code

Syntax

Each function is called and executed sequentially with the arguments being passed are result of the function itself.

If error is encountered, the main callback is called.

Example

I used async-waterfall to

1. Download files from an AWS S3 bucket.
2. Process the files
3. Upload the files back to another AWS S3

Since this is an async task, it waits at each

We can understand the code from the following, which is from AWS documentation.

Basically what is happening is, An object is downloaded from S3 bucket using download function, the resultant object is passed as an argument to the next function (Transform) which converts the object into a thumbnail. The thumbnail is passed as an argument to the upload function where in it is uploaded to another bucket.