Beanstalkd

beanstalkd

  • Mac OS X

  • Ubuntu

    • Installation Using Aptitude

      • Installation

        aptitude install -y beanstalkd

      • Version

        $ beanstalkd -v
beanstalkd 1.4.6
$ /usr/bin/beanstalkd -v
beanstalkd 1.4.6

      • Tips: Version 1.4.6 had an issue with delayed jobs

    • Installation From Source

      • Installation

        git clone https://github.com/kr/beanstalkd
cd beanstalkd
make
make install
# install -d /usr/local/bin/
# install beanstalkd /usr/local/bin/beanstalkd

      • Version

        $ /usr/local/bin/beanstalkd -v
beanstalkd 1.10+21+gb7b4a6a

      • Tips: If you want to use service beanstalkd xxx, you should first Installation Using Aptitude, then Installation From Source

      • /etc/init.d/beanstalkd

        # Default is ``/usr/bin/beanstalkd``
# You can change it to use other server's location
# DAEMON=/usr/bin/beanstalkd # Introduce the server's location here
DAEMON=/usr/local/bin/beanstalkd # Introduce the server's location here

    • Start

      # To start the service:
service beanstalkd start

      • Error

        $ service beanstalkd start
* Starting in-memory queueing server  beanstalkd
beanstalkd not configured to start, please edit /etc/default/beanstalkd to enable

      • /etc/default/beanstalkd

        ## Defaults for the beanstalkd init script, /etc/init.d/beanstalkd on
## Debian systems. Append ``-b /var/lib/beanstalkd'' for persistent
## storage.
BEANSTALKD_LISTEN_ADDR=0.0.0.0
BEANSTALKD_LISTEN_PORT=11300
DAEMON_OPTS="-l $BEANSTALKD_LISTEN_ADDR -p $BEANSTALKD_LISTEN_PORT"
## Uncomment to enable startup during boot.
START=yes

      • Start

        $ service beanstalkd start
* Starting in-memory queueing server  beanstalkd                                                     start-stop-daemon: unable to open pidfile '/var/run/beanstalkd.pid' for writing (Permission denied)
                                                                                              [fail]
$ sudo service beanstalkd start
* Starting in-memory queueing server  beanstalkd                                              [ OK ]

    • Stop

      # To stop the service:
service beanstalkd stop

    • Restart

      # To restart the service:
service beanstalkd restart

    • Status

      # To check the status:
service beanstalkd status

django-beanstalkd

  • Installation

    pip install -e git+https://github.com/jonasvp/django-beanstalkd.git#egg=django-beanstalkd

  • Settings.py

    • INSTALLED_APPS

      INSTALLED_APPS = (
    ...
    'django_beanstalkd',
    ...
)

    • BEANSTALK_SERVER

      # My beanstalkd server
BEANSTALK_SERVER = '127.0.0.1:11300'  # the default value, change 127.0.0.1 to ip

  • Register Jobs

  • Start Worker

    python manage.py beanstalk_worker -w 5 -l debug
    • One Server vs. Multi Server
      • If Multi Server may cause Multi Server get Same Job, lead to CommandFailed

  • Clients

    from django_beanstalkd import BeanstalkClient

client = BeanstalkClient()
client.call('beanstalk_example.background_counting', '5')

Example

  • INSTALLED_APPS

    INSTALLED_APPS = (
    ...
    'beanstalk_example',
    ...
)

  • python manage.py

    python manage.py beanstalk_example_client

Start Order

  • Beanstalkd ==> Worker ==> Django

Delayed Job

client.call('beanstalk_example.background_counting', '5', delay=120, ttr=TTR)

  • def call(self, func, arg='', priority=DEFAULT_PRIORITY, delay=0, ttr=DEFAULT_TTR):

    • priority: an integer number that specifies the priority. Jobs with a smaller priority get executed first

    • delay: how many seconds to wait before the job can be reserved

    • ttr: how many seconds a worker has to process the job before it gets requeued

    • DEFAULT_TTR

      In [1]: from beanstalkc import DEFAULT_TTR
In [2]: DEFAULT_TTR
Out[2]: 120

Logger

logger = logging.getLogger('django_beanstalkd')
logger.info('ooo')

Problems

  • Delayed Job Not Execute

  • SocketError

    • SocketError

      File "/home/diors/env/src/django-beanstalkd-master/django_beanstalkd/__init__.py", line 42, in call
    self._beanstalk.use(func)
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 190, in use
    return self._interact_value('use %s\r\n' % name, ['USING'])
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 110, in _interact_value
    return self._interact(command, expected_ok, expected_err)[0]
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 87, in _interact
    status, results = self._read_response()
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 98, in _read_response
    raise SocketError()
SocketError

    • SocketError: [Errno 104] Connection reset by peer

          File "/home/diors/env/src/django-beanstalkd-master/django_beanstalkd/__init__.py", line 42, in call
    self._beanstalk.use(func)
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 190, in use
    return self._interact_value('use %s\r\n' % name, ['USING'])
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 110, in _interact_value
    return self._interact(command, expected_ok, expected_err)[0]
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 87, in _interact
    status, results = self._read_response()
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 96, in _read_response
    line = SocketError.wrap(self._socket_file.readline)
  File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 43, in wrap
    raise SocketError(err)
SocketError: [Errno 104] Connection reset by peer

  • CommandFailed

      File "/home/diors/env/local/lib/python2.7/site-packages/beanstalkc.py", line 91, in _interact
    raise CommandFailed(command.split()[0], status, results)
beanstalkc.CommandFailed: ('delete', 'NOT_FOUND', [])

References

[1] beanstalkd@TT4IT, beanstalkd — Beanstalk is a simple, fast work queue

[2] beanstalkc@Github, beanstalkc — A simple beanstalkd client library for Python

[3] django-beanstalkd@Github, django-beanstalkd — A convenience wrapper for beanstalkd clients and workers in Django using the beanstalkc library for Python

[4] O.S. Tezer@DigitalOcean Community, How To Install and Use Beanstalkd Work Queue on a VPS

results matching ""

    No results matching ""