Tuesday, June 23. 2009
Doing the work elsewhere - Sidebar - running the worker
I'm taking a slight diversion now, to show you how the main worker processor runs. There are two parts to it - the actual worker, written in PHP, and the script that keeps running it.
For testing with return from the worker, we'll just return a random number. In order to avoid returning a normally used exit value, I've picked a few numbers for our controls, up around the 100 range. By default a 'die()' or 'exit' will return a '0', so we can't use that to act on - though we will use it as a fall-back as a generic error. Ideally, we won't get one, instead we want the code in all the workers to just run as planned, and then have the worker execute a planned restart - and we will just immediately restart. We may also choose to have the worker process specifically stop - and so we'll have an exit code for that. If there are any codes we don't understand, we'll slow the system down with a 'sleep()' to avoid running away with the process.
#!/usr/bin/php
<?php
// cli-beanstalk-worker.php
// for testing of the BASH script
exit (rand(95, 100));
/* normally we would return one of
# 97 - planned pause/restart
# 98 - planned restart
# 99 - planned stop, exit.
# anything else is an unplanned restart
*/
?>
The actual script that is run from the command line is a pretty simple BASH script - all it's got to do is to loop, until it gets a particular set of exit values back.
#!/bin/bash
# startBeanstalkd-worker.sh
# a shell script that keeps looping until an exit code is given
# if it's does an exit(0), restart after a second - or if it's a declared error
# if we've restarted in a planned fashion, we don't bother with any pause
# and for one particular code, exit the script entirely.
# The numbers 97, 98, 99 must match what is returned from the PHP script
nice php -q -f ./cli-beanstalk-worker.php -- $@
ERR=$?
## Possibilities
# 97 - planned pause/restart
# 98 - planned restart
# 99 - planned stop, exit.
# 0 - unplanned restart (as returned by "exit;")
# - Anything else is also unplanned paused/restart
if [ $ERR -eq 97 ]
then
# a planned pause, then restart
echo "97: PLANNED_PAUSE - wait 1";
sleep 1;
exec $0 $@;
fi
if [ $ERR -eq 98 ]
then
# a planned restart - instantly
echo "98: PLANNED_RESTART";
exec $0 $@;
fi
if [ $ERR -eq 99 ]
then
# planned complete exit
echo "99: PLANNED_SHUTDOWN";
exit 0;
fi
# unplanned exit, pause, and restart
echo "unplanned restart: err:" $ERR;
echo "sleeping for 1 sec"
sleep 1
exec $0 $@
So, if it's an exit value we know, we either
1/ pause, then restart
2/ immediately restart
3/ exit the loop.
If its any other value, we pause, and restart.
The bash command 'exec $0 $@' will re-run the current script ($0) with the original arguments ($@) - but with the 'exec', replaces the current process with a specified command. Normally, when the shell encounters a command, it forks off a child process to actually execute the command. Using the exec builtin, the shell does not fork, and the command exec'ed replaces the shell.
Save both the PHP and bash script, and tthen you can start the script with 'sh startBeanstalkd-worker.sh', run it a few times to see a lot of (deliberate) errors that cause the bash script to pause before restart, immediately restart and finally exit.
With this bash script in place, we can now run the script as many times as we need - and it will keep running, until we specifically tell it to exit. As usefully, we can exit the php worker, and have it execute a planned restart - which will clear any overheads that the script may have picked up with memory or resource allocation.
Next time, we'll put some simple tasks into the queue.
#!/usr/bin/php
<?php
// cli-beanstalk-worker.php
// for testing of the BASH script
exit (rand(95, 100));
/* normally we would return one of
# 97 - planned pause/restart
# 98 - planned restart
# 99 - planned stop, exit.
# anything else is an unplanned restart
*/
?>
#!/bin/bash
# startBeanstalkd-worker.sh
# a shell script that keeps looping until an exit code is given
# if it's does an exit(0), restart after a second - or if it's a declared error
# if we've restarted in a planned fashion, we don't bother with any pause
# and for one particular code, exit the script entirely.
# The numbers 97, 98, 99 must match what is returned from the PHP script
nice php -q -f ./cli-beanstalk-worker.php -- $@
ERR=$?
## Possibilities
# 97 - planned pause/restart
# 98 - planned restart
# 99 - planned stop, exit.
# 0 - unplanned restart (as returned by "exit;")
# - Anything else is also unplanned paused/restart
if [ $ERR -eq 97 ]
then
# a planned pause, then restart
echo "97: PLANNED_PAUSE - wait 1";
sleep 1;
exec $0 $@;
fi
if [ $ERR -eq 98 ]
then
# a planned restart - instantly
echo "98: PLANNED_RESTART";
exec $0 $@;
fi
if [ $ERR -eq 99 ]
then
# planned complete exit
echo "99: PLANNED_SHUTDOWN";
exit 0;
fi
# unplanned exit, pause, and restart
echo "unplanned restart: err:" $ERR;
echo "sleeping for 1 sec"
sleep 1
exec $0 $@
So, if it's an exit value we know, we either
1/ pause, then restart
2/ immediately restart
3/ exit the loop.
If its any other value, we pause, and restart.
The bash command 'exec $0 $@' will re-run the current script ($0) with the original arguments ($@) - but with the 'exec', replaces the current process with a specified command. Normally, when the shell encounters a command, it forks off a child process to actually execute the command. Using the exec builtin, the shell does not fork, and the command exec'ed replaces the shell.
Save both the PHP and bash script, and tthen you can start the script with 'sh startBeanstalkd-worker.sh', run it a few times to see a lot of (deliberate) errors that cause the bash script to pause before restart, immediately restart and finally exit.
With this bash script in place, we can now run the script as many times as we need - and it will keep running, until we specifically tell it to exit. As usefully, we can exit the php worker, and have it execute a planned restart - which will clear any overheads that the script may have picked up with memory or resource allocation.
Next time, we'll put some simple tasks into the queue.
Hi,
What kind of beanstalk client library do you use or have you written your own? Is it in PHP space or a C extension? Hehe I guess I will wait for your next post
Is there any way to run more the 1 worker (bash script) at a time?
Multiple workers running is not a problem – Beanstalkd will keep the jobs separate even if multiple ones are reserved.
You can run more than one worker at a time, by either putting them into the background with nohup, or running them in the foreground in ‘screen’. I’ve yet to find a good way to to autostart/run one or more workers though.