Tuesday, October 6. 2009
Doing the work elsewhere - Adding a job to the queue
I've previously shown you why you may want to put some tasks through a queuing system, what sort of jobs you could define, plus how to keep a worker process running for as long as you would like (but still be mindful of problems that happen).
In this post, I'll show you how to put the messages into the queue, and we'll also make a start on reading them back out.
In this post, I'll show you how to put the messages into the queue, and we'll also make a start on reading them back out.
For PHP, there are two BeanstalkD client libraries available.
Although I've previously used the first class in live code, I'm preferring the second, 'Pheanstalk', for this article. It is more regularly worked on, and uses object orientation to the fullest, plus it's got a test suite (based on Simpletest, which is included in the download).
Using it, according to the example is simple:
<?php
// register Pheanstalk class loader
require_once 'pheanstalk_init.php';
$pheanstalk = new Pheanstalk('127.0.0.1');
// ----------------------------------------
// producer (queues jobs)
$pheanstalk
->useTube('testtube')
->put("job payload goes here\n");
// ----------------------------------------
// worker (performs jobs)
$job = $pheanstalk->watch('testtube')
->ignore('default')
->reserve();
echo $job->getData();
$pheanstalk->delete($job);
?>The 'pheanstalk_init.php' file adds an autoloader, though you may find it advantageous to move the main class file hierarchy from where it had been downloaded into its own directory so that an existing (for example Zend Framework) auto-loader can find it.
As you see above, the Object Orientation lends itself well to (an optional) 'fluid' programming style, where an object is returned and then can be acted on in turn ("
So, putting simple data into the queue, is, well, simple (as it should be). There are advantages in wrapping this simplicity into our own class though. Some examples
Each may be simple enough to create a simple loop, but it might be advantagous to push that down into a class - and especially with the final idea.
How to store the meta-information then? It should be a text-friendly, but concise format, and quick to parse. Here, JSON (or the related Yaml) fits the bill quite nicely.
function putTask($data, $priority, $delay = 0, $tube = 'default', $canIgnore = false)
{
$taskInfo = array('task' => $data,
'_queuedAt' => time(),
'_ignorable'=> $canIgnore,
);
$json = json_encode($taskInfo);
$this->pheanstalk->useTube($tube)
->put($json, $priority, $delay, self::DEFAULT_TTR);
}
?>Processing it at the other end, after it has been fetched by the worker is a simple matter of running 'json_decode()' and extracting the ['task'] from the results before running it.
- BeanStalk.class.php - http://sourceforge.net/projects/beanstalk/
- Pheanstalk - http://github.com/pda/pheanstalk
Although I've previously used the first class in live code, I'm preferring the second, 'Pheanstalk', for this article. It is more regularly worked on, and uses object orientation to the fullest, plus it's got a test suite (based on Simpletest, which is included in the download).
Using it, according to the example is simple:
<?php
// register Pheanstalk class loader
require_once 'pheanstalk_init.php';
$pheanstalk = new Pheanstalk('127.0.0.1');
// ----------------------------------------
// producer (queues jobs)
$pheanstalk
->useTube('testtube')
->put("job payload goes here\n");
// ----------------------------------------
// worker (performs jobs)
$job = $pheanstalk->watch('testtube')
->ignore('default')
->reserve();
echo $job->getData();
$pheanstalk->delete($job);
?>
As you see above, the Object Orientation lends itself well to (an optional) 'fluid' programming style, where an object is returned and then can be acted on in turn ("
$pheanstalk->useTube('testtube')->put("job payload goes here\n");").So, putting simple data into the queue, is, well, simple (as it should be). There are advantages in wrapping this simplicity into our own class though. Some examples
- We want to put the same job into the queue multiple times - for example, a call to check some data in 1, 10 and 20 seconds time.
- Adding a new default priority - or with multiple classes, a small range of defaults
- Adding in other (meta) information about the job that is being run, such as when it was queued, and how important it is. Some tasks might be urgent, but not important - ie, if we have the opportunity, run them now - but it doesn't have to be run at all.
Each may be simple enough to create a simple loop, but it might be advantagous to push that down into a class - and especially with the final idea.
How to store the meta-information then? It should be a text-friendly, but concise format, and quick to parse. Here, JSON (or the related Yaml) fits the bill quite nicely.
function putTask($data, $priority, $delay = 0, $tube = 'default', $canIgnore = false)
{
$taskInfo = array('task' => $data,
'_queuedAt' => time(),
'_ignorable'=> $canIgnore,
);
$json = json_encode($taskInfo);
$this->pheanstalk->useTube($tube)
->put($json, $priority, $delay, self::DEFAULT_TTR);
}
?>
Comments
Display comments as
(Linear | Threaded)
Add Comment
