Beanstalkd
beanstalkd
Mac OS X
Installation
brew install beanstalkd
Version
$ beanstalkd -v beanstalkd 1.10
Script
Ubuntu
Installation Using Aptitude
Installation
aptitude install -y beanstalkd
Version
$ beanstalkd -v beanstalkd 1.4.6 $ /usr/bin/beanstalkd -v beanstalkd 1.4.6
Tips: Version 1.4.6 had an issue with delayed jobs
Installation From Source
Installation
git clone https://github.com/kr/beanstalkd cd beanstalkd make make install # install -d /usr/local/bin/ # install beanstalkd /usr/local/bin/beanstalkd
Version
$ /usr/local/bin/beanstalkd -v beanstalkd 1.10+21+gb7b4a6a
Tips: If you want to use
service beanstalkd xxx
, you should firstInstallation Using Aptitude
, thenInstallation From Source
/etc/init.d/beanstalkd
# Default is ``/usr/bin/beanstalkd`` # You can change it to use other server's location # DAEMON=/usr/bin/beanstalkd # Introduce the server's location here DAEMON=/usr/local/bin/beanstalkd # Introduce the server's location here
Start
# To start the service: service beanstalkd start
Error
$ service beanstalkd start * Starting in-memory queueing server beanstalkd beanstalkd not configured to start, please edit /etc/default/beanstalkd to enable
/etc/default/beanstalkd
## Defaults for the beanstalkd init script, /etc/init.d/beanstalkd on ## Debian systems. Append ``-b /var/lib/beanstalkd'' for persistent ## storage. BEANSTALKD_LISTEN_ADDR=0.0.0.0 BEANSTALKD_LISTEN_PORT=11300 DAEMON_OPTS="-l $BEANSTALKD_LISTEN_ADDR -p $BEANSTALKD_LISTEN_PORT" ## Uncomment to enable startup during boot. START=yes
Start
$ service beanstalkd start * Starting in-memory queueing server beanstalkd start-stop-daemon: unable to open pidfile '/var/run/beanstalkd.pid' for writing (Permission denied) [fail] $ sudo service beanstalkd start * Starting in-memory queueing server beanstalkd [ OK ]
Stop
# To stop the service: service beanstalkd stop
Restart
# To restart the service: service beanstalkd restart
Status
# To check the status: service beanstalkd status
django-beanstalkd
Installation
pip install -e git+https://github.com/jonasvp/django-beanstalkd.git#egg=django-beanstalkd
Settings.py
INSTALLED_APPS
INSTALLED_APPS = ( ... 'django_beanstalkd', ... )
BEANSTALK_SERVER
# My beanstalkd server BEANSTALK_SERVER = '127.0.0.1:11300' # the default value, change 127.0.0.1 to ip
Register Jobs
- beanstalk_jobs.py
- Create
beanstalk_jobs.py
under Django APP's dir - Use
django_beanstalkd.beanstalk_job
as decorator - Example Beanstalk Job File
- Create
- Warning
- The job must accept a single string argument as passed by the caller
- Pass Multi Arguments
json.dumps
&json.loads
- beanstalk_jobs.py
Start Worker
python manage.py beanstalk_worker -w 5 -l debug
One Server
vs.Multi Server
- If
Multi Server
may causeMulti Server get Same Job
, lead toCommandFailed
- If
Clients
from django_beanstalkd import BeanstalkClient client = BeanstalkClient() client.call('beanstalk_example.background_counting', '5')
Example
INSTALLED_APPS
INSTALLED_APPS = ( ... 'beanstalk_example', ... )
python manage.py
python manage.py beanstalk_example_client
Start Order
- Beanstalkd ==> Worker ==> Django
Delayed Job
client.call('beanstalk_example.background_counting', '5', delay=120, ttr=TTR)
def call(self, func, arg='', priority=DEFAULT_PRIORITY, delay=0, ttr=DEFAULT_TTR):
priority: an integer number that specifies the priority. Jobs with a smaller priority get executed first
delay: how many seconds to wait before the job can be reserved
ttr: how many seconds a worker has to process the job before it gets requeued
DEFAULT_TTR
In [1]: from beanstalkc import DEFAULT_TTR In [2]: DEFAULT_TTR Out[2]: 120
Logger
logger = logging.getLogger('django_beanstalkd')
logger.info('ooo')
Problems
Delayed Job Not Execute
SocketError
SocketError
File "/home/diors/env/src/django-beanstalkd-master/django_beanstalkd/__init__.py", line 42, in call self._beanstalk.use(func) File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 190, in use return self._interact_value('use %s\r\n' % name, ['USING']) File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 110, in _interact_value return self._interact(command, expected_ok, expected_err)[0] File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 87, in _interact status, results = self._read_response() File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 98, in _read_response raise SocketError() SocketError
SocketError: [Errno 104] Connection reset by peer
File "/home/diors/env/src/django-beanstalkd-master/django_beanstalkd/__init__.py", line 42, in call self._beanstalk.use(func) File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 190, in use return self._interact_value('use %s\r\n' % name, ['USING']) File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 110, in _interact_value return self._interact(command, expected_ok, expected_err)[0] File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 87, in _interact status, results = self._read_response() File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 96, in _read_response line = SocketError.wrap(self._socket_file.readline) File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 43, in wrap raise SocketError(err) SocketError: [Errno 104] Connection reset by peer
CommandFailed
File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 91, in _interact raise CommandFailed(command.split()[0], status, results) beanstalkc.CommandFailed: ('delete', 'NOT_FOUND', [])
References
[1] beanstalkd@TT4IT, beanstalkd — Beanstalk is a simple, fast work queue
[2] beanstalkc@Github, beanstalkc — A simple beanstalkd client library for Python
[3] django-beanstalkd@Github, django-beanstalkd — A convenience wrapper for beanstalkd clients and workers in Django using the beanstalkc library for Python
[4] O.S. Tezer@DigitalOcean Community, How To Install and Use Beanstalkd Work Queue on a VPS