May 17, 2016

Queue Jobs with Kue

By Jscrambler | 5 min read

queue_jobs_with_kue

Ask developers about Redis and most will tell you that it’s a key/value store. However it’s more than that; it’s also a very useful mechanism for implementing job queues.

There are a number of advantages for encapsulating units of work as jobs:

  • A job can be run in the background. In the context of a web application, that means you can handle the request without waiting for the job to complete.
  • You can delay the execution of a job, setting it aside to run at a later date or time.
  • A job can run in a different context to your “main” application, be that a different process or thread, environment or even a different server.
  • If a job fails, you can configure your application to re-try it multiple times.
  • You can also prioritise jobs. Suppose you need to email a failure notice as a matter of urgency, while your application is sending out a bunch of newsletters. Set a high priority and it will “jump to the front of the queue”, so to speak.
  • Many queue implementations also provide the ability to monitor jobs; their status – whether they’ve succeeded or failed, or are waiting to be re-tried, for example – as well as to monitor the progress of long-running jobs.

Now that we’ve looked at some of the advantages of jobs, let’s look at one of the ways you can implement it in your Node.js applications; using Kue.

Introducing Kue

Kue is a Node.js package that provides a level of abstraction over Redis queues, as well as providing some useful tools for monitoring your jobs.

At its most basic, Kue allows you to define a unit of code as a job and behind the scenes it will place the necessary information – what to run, and the data to run it with – on a Redis queue. A worker is then responsible for actually running those jobs.

Pre-requisites

To use Kue you’ll need Node.js and npm installed, and of course it requires that you have Redis installed and configured. We’ll assume you have Redis installed on localhost (127.0.0.1) and listening on the default port – 6379 – but you can of course configure this appropriately.

Installation

Install Kue using npm:

npm install kue

Then you’ll need to require it:

var kue = require('kue');

Creating a Job

Before you create a job, you need to create a queue instance:

var queue = kue.createQueue();

There are also a number of additional options such as naming queues and overriding the default Redis connection settings; consult the documentation for more details.

Now let’s create a job by calling the create() method on your newly-created queue. The first argument identifies the type of job, and any additional data is passed as a hash via the second argument. The job must then be saved to the queue. Here’s an example:

queue.create('email', {
    title: 'Welcome to the site',
    to: '[email protected]',
    template: 'welcome-email'
}).save();

Setting the Priority

Queues are set to “normal” priority by default, which means that they’ll be handled in a standard FIFO (First In First Out) manner. As you’d expect, a higher priority means the job will jump up the queue, and the reverse is true for low-priority jobs.

Priorities are defined by number or name; a name is simply a constant which maps to a numeric value, as follows:

Constant Value
low 10
normal 0
medium -5
high -10
critical -15

Do be aware that somewhat counterintuitively the lower the value, the higher the priority.

Setting the priority is easy; just call the priority() method on the job, before you save it. Either provide a numeric value:

queue.create(..).priority(-10).save();

…or, a string, referring to the table above:

queue.create(..).priority('high').save();

Try Jscrambler For Free

Re-trying Failed Jobs

By default, if a job fails then then it won’t be re-run. However a job will be re-tried if you provide the maximum number of attempts as follows:

queue.create(..).attempts(5).save();

Putting it Altogether

Since the priority(), attempts() and save() methods are all chainable, we can put it all together like this:

queue.create('email', {
    title: 'Welcome to the site',
    to: '[email protected]',
    template: 'welcome-email'
}).priority('high').attempts(5).save();

Running Jobs with a Worker

To run jobs which have been added to a queue, typically you’ll create a separate worker process. This should create a queue instance as before and call the process() method. The first argument should be the name of the job, and the second a function, which provides as arguments the job itself and a callback. For example:

var queue = kue.createQueue();

queue.process('email', function(job, done) {
    sendEmail(job.data, done);
});

You’ll notice from the example above that the data we provided when we added the job to the queue is available via the data property on the job object.

As is commonplace in a Node.js application, the done callback should be called with an Error as its first argument to indicate failure, or null otherwise. If you need to provide a result of the job – for example a status code, or the ID of a database entry – then you can provide it as an entirely optional second argument.

Additional Considerations

There are a couple of things you may wish to consider when creating a worker.

To run across multiple processes, you may wish to consider using Cluster.

As you might have guessed, it’s important that the worker remains running at all times; for this you might wish to consider either forever or PM2. You might also find supervisord useful for ensuring the constant running of the process, even after a system restart.

Monitoring Jobs

In addition to providing a method for creating and processing jobs, Kue also offers a number of features for monitoring them.

The UI

Kue provides an optional, very simple user interface for monitoring jobs as an Express application.

You can either run it as a standalone application from the command-line:

node_modules / kue / bin / kue - dashboard - p 3050 - r redis: //127.0.0.1:3000

Or from your own Express application:

kue.app.listen(3000);

Alternatively, the Kue UI package provides a slicker, more comprehensive UI. You can see it action in the screenshot below:

The Kue UI package in action

The JSON API

For even more flexibility for monitoring jobs, Kue also provides a JSON API.

Summary

Jobs are great way of adding fault tolerance, decoupling your application’s architecture and for executing long-running processes in the background. Kue not only provides a high-level abstraction to Redis queues for just that, but also a suite of tools for monitoring the status of all of your jobs. You can find out more on the project page and in the documentation.

As a final word, don't forget to download our free data sheet on JavaScript Security Threats, which provides an overview of the most relevant attacks and how to prevent them.

Author
JscramblerThe leader in client-side Web security. With Jscrambler, JavaScript applications become self-defensive and capable of detecting and blocking client-side attacks like Magecart.
View All Posts

Subscribe to our weekly newsletter

Learn more about new security threats and technologies.

I agree to receive these emails and accept the Privacy Policy.