Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 46 additions & 39 deletions content/pages/07-web-development/26-task-queues.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ HTTP request-response cycle.


## Why are task queues necessary?
Tasks are handled asynchronously either because they are not initiated by
Tasks are handled asynchronously either because they are not initiated by
an HTTP request or because they are long-running jobs that would dramatically
reduce the performance of an HTTP response.

Expand All @@ -27,12 +27,12 @@ request-response cycle. The query could be performed in the background on a
fixed interval with the results stored in the database. When an
HTTP request comes in that needs those results a query would simply fetch the
precalculated result instead of re-executing the longer query.
This precalculation scenario is a form of [caching](/caching.html) enabled
This precalculation scenario is a form of [caching](/caching.html) enabled
by task queues.

Other types of jobs for task queues include

* spreading out large numbers of independent database inserts over time
* spreading out large numbers of independent database inserts over time
instead of inserting everything at once

* aggregating collected data values on a fixed interval, such as every
Expand All @@ -42,39 +42,42 @@ Other types of jobs for task queues include


## Task queue projects
The defacto standard Python task queue is [Celery](/celery.html). The other
task queue projects that arise tend to come from the perspective that
Celery is overly complicated for simple use cases. My recommendation is to
put the effort into Celery's reasonable learning curve as it is worth the
The defacto standard Python task queue is [Celery](/celery.html). The other
task queue projects that arise tend to come from the perspective that
Celery is overly complicated for simple use cases. My recommendation is to
put the effort into Celery's reasonable learning curve as it is worth the
time it takes to understand how to use the project.

* The [Celery](/celery.html) distributed task queue is the
most commonly used Python library for handling asynchronous tasks and
most commonly used Python library for handling asynchronous tasks and
scheduling.

* The [RQ (Redis Queue)](/redis-queue-rq.html) is a simple Python
library for queueing jobs and processing them in the background with
workers. RQ is backed by Redis and is designed to have a low barrier to
library for queueing jobs and processing them in the background with
workers. RQ is backed by Redis and is designed to have a low barrier to
entry.

* [Taskmaster](https://github.com/dcramer/taskmaster) is a lightweight simple
distributed queue for handling large volumes of one-off tasks.
distributed queue for handling large volumes of one-off tasks.

* [Huey](http://huey.readthedocs.org/en/latest/) is a Redis-based task
queue that aims to provide a simple, yet flexible framework for
executing tasks. Huey supports task scheduling, crontab-like repeating
* [Huey](http://huey.readthedocs.org/en/latest/) is a Redis-based task
queue that aims to provide a simple, yet flexible framework for
executing tasks. Huey supports task scheduling, crontab-like repeating
tasks, result storage and automatic retry in the event of failure.

* [Kuyruk](https://kuyruk.readthedocs.io) is simple and easy to use task queue
system built on top of RabbitMQ. Although feature set is small, new features
can be added by extensions.

* [Dramatiq](https://dramatiq.io) is a fast and reliable alternative
to Celery. It supports RabbitMQ and Redis as message brokers.


## Hosted message and task queue services
Task queue third party services aim to solve the complexity issues that arise
when scaling out a large deployment of distributed task queues.

* [Iron.io](http://www.iron.io/) is a distributed messaging service platform
* [Iron.io](http://www.iron.io/) is a distributed messaging service platform
that works with many types of task queues such as Celery. It also is built
to work with other IaaS and PaaS environments such as Amazon Web Services
and Heroku.
Expand All @@ -84,27 +87,32 @@ when scaling out a large deployment of distributed task queues.
messages.

* [CloudAMQP](http://www.cloudamqp.com/) is at its core managed servers with
RabbitMQ installed and configured. This service is an option if you are
using RabbitMQ and do not want to maintain RabbitMQ installations on your
RabbitMQ installed and configured. This service is an option if you are
using RabbitMQ and do not want to maintain RabbitMQ installations on your
own servers.


## Open source examples that use task queues
* Take a look at the code in this open source
[Flask application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/flask)
and
[this Django application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/django)
* Take a look at the code in this open source
[Flask application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/flask)
and
[this Django application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/django)
for examples of how to use and deploy Celery with a Redis broker to
send text messages with these frameworks.
send text messages with these frameworks.

* [flask-celery-example](https://github.com/thrisp/flask-celery-example) is
a simple Flask application with Celery as a task queue and Redis as
the broker.

* [django_dramatiq_example](https://github.com/Bogdanp/django_dramatiq_example) and
[flask_dramatiq_example](https://github.com/Bogdanp/flask_dramatiq_example)
are simple apps that demo how you can use Dramatiq with Django and
Flask, respectively.


## Task queue resources
* [International Space Station notifications with Python and Redis Queue (RQ)](https://www.twilio.com/blog/2015/11/international-space-station-notifications-with-python-redis-queue-and-twilio-copilot.html)
shows how to combine the RQ task queue library with Flask to send
shows how to combine the RQ task queue library with Flask to send
text message notifications every time a condition is met - in this blog
post's case that the ISS is currently flying over your location on
Earth.
Expand All @@ -117,10 +125,10 @@ when scaling out a large deployment of distributed task queues.
short summaries for each one. The task queues are not all compatible with
Python but ones that work with it are tagged with the "Python" keyword.

* [Why Task Queues](http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619)
is a presentation for what task queues are and why they are needed.
* [Why Task Queues](http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619)
is a presentation for what task queues are and why they are needed.

* [Asynchronous Processing in Web Applications Part One](http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/)
* [Asynchronous Processing in Web Applications Part One](http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/)
and [Part Two](http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/)
are great reads for understanding the difference between a task queue and
why you shouldn't use your database as one.
Expand All @@ -130,18 +138,18 @@ when scaling out a large deployment of distributed task queues.
provides a detailed walkthrough of setting up workers to use RQ with
Redis.

* Heroku has a clear walkthrough for using
* Heroku has a clear walkthrough for using
[RQ for background tasks](https://devcenter.heroku.com/articles/python-rq).

* [How to use Celery with RabbitMQ](https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps)
is a detailed walkthrough for using these tools on an Ubuntu VPS.

* [Celery - Best Practices](https://denibertovic.com/posts/celery-best-practices/)
explains things you should not do with Celery and shows some underused
explains things you should not do with Celery and shows some underused
features for making task queues easier to work with.

* [Celery in Production](http://www.caktusgroup.com/blog/2014/09/29/celery-production/)
on the Caktus Group blog contains good practices from their experience
on the Caktus Group blog contains good practices from their experience
using Celery with RabbitMQ, monitoring tools and other aspects not often
discussed in existing documentation.

Expand All @@ -151,25 +159,25 @@ when scaling out a large deployment of distributed task queues.
* This [Celery tasks checklist](http://celerytaskschecklist.com/) has
some nice tips and resources for using Celery in your applications.

* Heroku wrote about how to
* Heroku wrote about how to
[secure Celery](https://engineering.heroku.com/blogs/2014-09-15-securing-celery)
when tasks are otherwise sent over unencrypted networks.

* Miguel Grinberg wrote a nice post on using the
[task queue Celery with Flask](http://blog.miguelgrinberg.com/post/using-celery-with-flask).
* Miguel Grinberg wrote a nice post on using the
[task queue Celery with Flask](http://blog.miguelgrinberg.com/post/using-celery-with-flask).
He gives an overview of Celery followed by specific code to set up the task
queue and integrate it with Flask.

* [3 Gotchas for Working with Celery](https://wiredcraft.com/blog/3-gotchas-for-celery/)
are things to keep in mind when you're new to the Celery task queue
are things to keep in mind when you're new to the Celery task queue
implementation.

* [Deferred Tasks and Scheduled Jobs with Celery 3.1, Django 1.7 and Redis](https://godjango.com/63-deferred-tasks-and-scheduled-jobs-with-celery-31-django-17-and-redis/)
is a video along with code that shows how to set up Celery with Redis as the
broker in a Django application.

* [Setting up an asynchronous task queue for Django using Celery and Redis](http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/)
is a straightforward tutorial for setting up the Celery task queue for
is a straightforward tutorial for setting up the Celery task queue for
Django web applications using the Redis broker on the back end.

* [Three quick tips from two years with Celery](https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb)
Expand All @@ -178,21 +186,20 @@ when scaling out a large deployment of distributed task queues.


## Task queue learning checklist
1. Pick a slow function in your project that is called during an HTTP
1. Pick a slow function in your project that is called during an HTTP
request.

1. Determine if you can precompute the results on a fixed interval instead
1. Determine if you can precompute the results on a fixed interval instead
of during the HTTP request. If so, create a separate function you can call
from elsewhere then store the precomputed value in the database.

1. Read the Celery documentation and the links in the resources section below
to understand how the project works.

1. Install a message broker such as RabbitMQ or Redis and then add Celery to
1. Install a message broker such as RabbitMQ or Redis and then add Celery to
your project. Configure Celery to work with the installed message broker.

1. Use Celery to invoke the function from step one on a regular basis.

1. Have the HTTP request function use the precomputed value instead of the
1. Have the HTTP request function use the precomputed value instead of the
slow running code it originally relied upon.

35 changes: 35 additions & 0 deletions content/pages/07-web-development/40-dramatiq.markdown
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
title: Dramatiq
category: page
slug: dramatiq
sortorder: 0730
toc: False
sidebartitle: Dramatiq
meta: Dramatiq is a fast and reliable alternative to Celery. It supports RabbitMQ and Redis as message brokers.


# Dramatiq

[Dramatiq][dramatiq] is a fast and reliable Python [task queue][taskqueues]
that came about as an alternative to Celery. It supports RabbitMQ and
Redis as message brokers.

<div class="well see-also">Dramatiq is an implementation of the <a href="/task-queues.html">task queue</a> concept. Learn more in the <a href="/web-development.html">web development</a> chapter or view the <a href="/table-of-contents.html">table of contents</a> for all topics.</div>

## Dramatiq resources

* The [Dramatiq][dramatiq] documentation website.
* The [source code][dramatiq-source].
* [django_dramatiq][django_dramatiq] is a Django app for integrating
Django and Dramatiq.
* [django_dramatiq_example][django_dramatiq_example] is a simple
Django app demoing how you can integrate Django and Dramatiq.
* [flask_dramatiq_example][flask_dramatiq_example] is a simple app
demoing how you can integrate Flask and Dramatiq.


[taskqueues]: /task-queues.html
[dramatiq]: https://dramatiq.io
[dramatiq-source]: https://github.com/Bogdanp/dramatiq
[django_dramatiq]: https://github.com/Bogdanp/django_dramatiq
[django_dramatiq_example]: https://github.com/Bogdanp/django_dramatiq_example
[flask_dramatiq_example]: https://github.com/Bogdanp/flask_dramatiq_example
18 changes: 18 additions & 0 deletions theme/templates/choices/dramatiq.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
<h3>Do you want to learn more about task queues, or another topic?</h3>
<div class="row">
<div class="col-md-4">
<div class="well select-next">
{% include "choices/buttons/task-queues.html" %}
</div>
</div>
<div class="col-md-4">
<div class="well select-next">
{% include "choices/buttons/deployment.html" %}
</div>
</div>
<div class="col-md-4">
<div class="well select-next">
{% include "choices/buttons/logging.html" %}
</div>
</div>
</div>
1 change: 1 addition & 0 deletions theme/templates/table-of-contents.html
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ <h4 class="toc-subsection">7.5 <a href="/javascript.html">JavaScript</a></h4>
<h4 class="toc-subsection">7.6 <a href="/task-queues.html">Task queues</a></h4>
<div class="toc"><a href="/celery.html">Celery</a></div>
<div class="toc"><a href="/redis-queue-rq.html">Redis Queue (RQ)</a></div>
<div class="toc"><a href="/dramatiq.html">Dramatiq</a></div>
<div class="toc soon">Huey</div>

<h4 class="toc-subsection">7.7 <a href="/static-site-generator.html">Static site generators</a></h4>
Expand Down