Basic django, celery and rabbitmq example.
Do you know what I don’t like? When repos have examples, but they don’t contain all of the source files. Most of the times I learn the best when I can see all the source files of a very basic example so that I can get a grip of how it all fits together. Most of what I’m saying here has been done before, but it was disparate, so I slapped it together. Nothing fancy, and hopefully it helps someone. It is mainly (almost entirely) based on django-celery’s example: found here. https://github.com/ask/django-celery/tree/master/examples. However that example doesn’t give examples on how to run it all (and uses your database as message broker).
Assumptions: Knowing how Django works, setting up an app, etc.
Why?
I’m working on a new side project (as I always do) with a few friends and we are going to need a simple job processor to run some image processing in the background. There are some great tools to do this, and I’m not quite equipped to say which is better (yet). I found celery and rabbitmq to do the job. Celery handles task processing with rabbitmq as the message broker. With celery you can use databases such as mongodb or redis. (NOTE: I don’t think it works with sqlite).
So. In order to process jobs through django, 3 processes need to run: rabbitmq, celery and the django server.
Rabbitmq:
On OS X, this is quite easy to install. I used homebrew (brew install rabbitmq). I had problems with it assigning symlinks, so I had to fix up some permissions.
This is also to install. On OS X, you simply run “easy_install django-celery”. You can also use pip.
Important django parts:
settings.py
Add “djcelery” to installed apps, and add:
import djcelery
djcelery.setup_uploader()
at the top. followed by:
BROKER_URL = “amqp://guest:guest@localhost:5672//”
This line that celery uses to connect to the specific messaging broker (in this case rabbitmq). It uses the guest user and password. If you want to use your own user/password combo, you should create that with rabbitmq.
tasks.py (some tasks)
and then in views.py, you can do something basic like this:
It adds three tasks that sleeps for 10 seconds each. All will finish at the same time.
To run it all:
1) Clone from github
2) Run rabbitmq: “rabbitmq-server” (just that. Type it in terminal)
3) Syncdb: “python manage.py syncdb”
4) Run django-celery: “python manage.py celeryd —log-level=info”
5) Run the django server: “python manage.py runserver” (or gunicorn, whatever you are using).
I might be blackboxing some parts, so if anyone needs some questions answered, just ask.
