From 8cb8cf0254b2d32af6ae18fd208e3724374fd7a5 Mon Sep 17 00:00:00 2001 From: Bogdan Popa Date: Sun, 5 Nov 2017 19:01:33 +0200 Subject: [PATCH] add Dramatiq --- .../26-task-queues.markdown | 85 ++++++++++--------- .../07-web-development/40-dramatiq.markdown | 35 ++++++++ theme/templates/choices/dramatiq.html | 18 ++++ theme/templates/table-of-contents.html | 1 + 4 files changed, 100 insertions(+), 39 deletions(-) create mode 100644 content/pages/07-web-development/40-dramatiq.markdown create mode 100644 theme/templates/choices/dramatiq.html diff --git a/content/pages/07-web-development/26-task-queues.markdown b/content/pages/07-web-development/26-task-queues.markdown index 9bad5e8fd..d7a2fd070 100644 --- a/content/pages/07-web-development/26-task-queues.markdown +++ b/content/pages/07-web-development/26-task-queues.markdown @@ -13,7 +13,7 @@ HTTP request-response cycle. ## Why are task queues necessary? -Tasks are handled asynchronously either because they are not initiated by +Tasks are handled asynchronously either because they are not initiated by an HTTP request or because they are long-running jobs that would dramatically reduce the performance of an HTTP response. @@ -27,12 +27,12 @@ request-response cycle. The query could be performed in the background on a fixed interval with the results stored in the database. When an HTTP request comes in that needs those results a query would simply fetch the precalculated result instead of re-executing the longer query. -This precalculation scenario is a form of [caching](/caching.html) enabled +This precalculation scenario is a form of [caching](/caching.html) enabled by task queues. Other types of jobs for task queues include -* spreading out large numbers of independent database inserts over time +* spreading out large numbers of independent database inserts over time instead of inserting everything at once * aggregating collected data values on a fixed interval, such as every @@ -42,39 +42,42 @@ Other types of jobs for task queues include ## Task queue projects -The defacto standard Python task queue is [Celery](/celery.html). The other -task queue projects that arise tend to come from the perspective that -Celery is overly complicated for simple use cases. My recommendation is to -put the effort into Celery's reasonable learning curve as it is worth the +The defacto standard Python task queue is [Celery](/celery.html). The other +task queue projects that arise tend to come from the perspective that +Celery is overly complicated for simple use cases. My recommendation is to +put the effort into Celery's reasonable learning curve as it is worth the time it takes to understand how to use the project. * The [Celery](/celery.html) distributed task queue is the - most commonly used Python library for handling asynchronous tasks and + most commonly used Python library for handling asynchronous tasks and scheduling. * The [RQ (Redis Queue)](/redis-queue-rq.html) is a simple Python - library for queueing jobs and processing them in the background with - workers. RQ is backed by Redis and is designed to have a low barrier to + library for queueing jobs and processing them in the background with + workers. RQ is backed by Redis and is designed to have a low barrier to entry. * [Taskmaster](https://github.com/dcramer/taskmaster) is a lightweight simple - distributed queue for handling large volumes of one-off tasks. + distributed queue for handling large volumes of one-off tasks. -* [Huey](http://huey.readthedocs.org/en/latest/) is a Redis-based task - queue that aims to provide a simple, yet flexible framework for - executing tasks. Huey supports task scheduling, crontab-like repeating +* [Huey](http://huey.readthedocs.org/en/latest/) is a Redis-based task + queue that aims to provide a simple, yet flexible framework for + executing tasks. Huey supports task scheduling, crontab-like repeating tasks, result storage and automatic retry in the event of failure. * [Kuyruk](https://kuyruk.readthedocs.io) is simple and easy to use task queue system built on top of RabbitMQ. Although feature set is small, new features can be added by extensions. +* [Dramatiq](https://dramatiq.io) is a fast and reliable alternative + to Celery. It supports RabbitMQ and Redis as message brokers. + ## Hosted message and task queue services Task queue third party services aim to solve the complexity issues that arise when scaling out a large deployment of distributed task queues. -* [Iron.io](http://www.iron.io/) is a distributed messaging service platform +* [Iron.io](http://www.iron.io/) is a distributed messaging service platform that works with many types of task queues such as Celery. It also is built to work with other IaaS and PaaS environments such as Amazon Web Services and Heroku. @@ -84,27 +87,32 @@ when scaling out a large deployment of distributed task queues. messages. * [CloudAMQP](http://www.cloudamqp.com/) is at its core managed servers with - RabbitMQ installed and configured. This service is an option if you are - using RabbitMQ and do not want to maintain RabbitMQ installations on your + RabbitMQ installed and configured. This service is an option if you are + using RabbitMQ and do not want to maintain RabbitMQ installations on your own servers. ## Open source examples that use task queues -* Take a look at the code in this open source - [Flask application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/flask) - and - [this Django application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/django) +* Take a look at the code in this open source + [Flask application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/flask) + and + [this Django application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/django) for examples of how to use and deploy Celery with a Redis broker to - send text messages with these frameworks. + send text messages with these frameworks. * [flask-celery-example](https://github.com/thrisp/flask-celery-example) is a simple Flask application with Celery as a task queue and Redis as the broker. +* [django_dramatiq_example](https://github.com/Bogdanp/django_dramatiq_example) and + [flask_dramatiq_example](https://github.com/Bogdanp/flask_dramatiq_example) + are simple apps that demo how you can use Dramatiq with Django and + Flask, respectively. + ## Task queue resources * [International Space Station notifications with Python and Redis Queue (RQ)](https://www.twilio.com/blog/2015/11/international-space-station-notifications-with-python-redis-queue-and-twilio-copilot.html) - shows how to combine the RQ task queue library with Flask to send + shows how to combine the RQ task queue library with Flask to send text message notifications every time a condition is met - in this blog post's case that the ISS is currently flying over your location on Earth. @@ -117,10 +125,10 @@ when scaling out a large deployment of distributed task queues. short summaries for each one. The task queues are not all compatible with Python but ones that work with it are tagged with the "Python" keyword. -* [Why Task Queues](http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619) - is a presentation for what task queues are and why they are needed. +* [Why Task Queues](http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619) + is a presentation for what task queues are and why they are needed. -* [Asynchronous Processing in Web Applications Part One](http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/) +* [Asynchronous Processing in Web Applications Part One](http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/) and [Part Two](http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/) are great reads for understanding the difference between a task queue and why you shouldn't use your database as one. @@ -130,18 +138,18 @@ when scaling out a large deployment of distributed task queues. provides a detailed walkthrough of setting up workers to use RQ with Redis. -* Heroku has a clear walkthrough for using +* Heroku has a clear walkthrough for using [RQ for background tasks](https://devcenter.heroku.com/articles/python-rq). * [How to use Celery with RabbitMQ](https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps) is a detailed walkthrough for using these tools on an Ubuntu VPS. * [Celery - Best Practices](https://denibertovic.com/posts/celery-best-practices/) - explains things you should not do with Celery and shows some underused + explains things you should not do with Celery and shows some underused features for making task queues easier to work with. * [Celery in Production](http://www.caktusgroup.com/blog/2014/09/29/celery-production/) - on the Caktus Group blog contains good practices from their experience + on the Caktus Group blog contains good practices from their experience using Celery with RabbitMQ, monitoring tools and other aspects not often discussed in existing documentation. @@ -151,17 +159,17 @@ when scaling out a large deployment of distributed task queues. * This [Celery tasks checklist](http://celerytaskschecklist.com/) has some nice tips and resources for using Celery in your applications. -* Heroku wrote about how to +* Heroku wrote about how to [secure Celery](https://engineering.heroku.com/blogs/2014-09-15-securing-celery) when tasks are otherwise sent over unencrypted networks. -* Miguel Grinberg wrote a nice post on using the - [task queue Celery with Flask](http://blog.miguelgrinberg.com/post/using-celery-with-flask). +* Miguel Grinberg wrote a nice post on using the + [task queue Celery with Flask](http://blog.miguelgrinberg.com/post/using-celery-with-flask). He gives an overview of Celery followed by specific code to set up the task queue and integrate it with Flask. * [3 Gotchas for Working with Celery](https://wiredcraft.com/blog/3-gotchas-for-celery/) - are things to keep in mind when you're new to the Celery task queue + are things to keep in mind when you're new to the Celery task queue implementation. * [Deferred Tasks and Scheduled Jobs with Celery 3.1, Django 1.7 and Redis](https://godjango.com/63-deferred-tasks-and-scheduled-jobs-with-celery-31-django-17-and-redis/) @@ -169,7 +177,7 @@ when scaling out a large deployment of distributed task queues. broker in a Django application. * [Setting up an asynchronous task queue for Django using Celery and Redis](http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/) - is a straightforward tutorial for setting up the Celery task queue for + is a straightforward tutorial for setting up the Celery task queue for Django web applications using the Redis broker on the back end. * [Three quick tips from two years with Celery](https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb) @@ -178,21 +186,20 @@ when scaling out a large deployment of distributed task queues. ## Task queue learning checklist -1. Pick a slow function in your project that is called during an HTTP +1. Pick a slow function in your project that is called during an HTTP request. -1. Determine if you can precompute the results on a fixed interval instead +1. Determine if you can precompute the results on a fixed interval instead of during the HTTP request. If so, create a separate function you can call from elsewhere then store the precomputed value in the database. 1. Read the Celery documentation and the links in the resources section below to understand how the project works. -1. Install a message broker such as RabbitMQ or Redis and then add Celery to +1. Install a message broker such as RabbitMQ or Redis and then add Celery to your project. Configure Celery to work with the installed message broker. 1. Use Celery to invoke the function from step one on a regular basis. -1. Have the HTTP request function use the precomputed value instead of the +1. Have the HTTP request function use the precomputed value instead of the slow running code it originally relied upon. - diff --git a/content/pages/07-web-development/40-dramatiq.markdown b/content/pages/07-web-development/40-dramatiq.markdown new file mode 100644 index 000000000..d47aa7463 --- /dev/null +++ b/content/pages/07-web-development/40-dramatiq.markdown @@ -0,0 +1,35 @@ +title: Dramatiq +category: page +slug: dramatiq +sortorder: 0730 +toc: False +sidebartitle: Dramatiq +meta: Dramatiq is a fast and reliable alternative to Celery. It supports RabbitMQ and Redis as message brokers. + + +# Dramatiq + +[Dramatiq][dramatiq] is a fast and reliable Python [task queue][taskqueues] +that came about as an alternative to Celery. It supports RabbitMQ and +Redis as message brokers. + +
Dramatiq is an implementation of the task queue concept. Learn more in the web development chapter or view the table of contents for all topics.
+ +## Dramatiq resources + +* The [Dramatiq][dramatiq] documentation website. +* The [source code][dramatiq-source]. +* [django_dramatiq][django_dramatiq] is a Django app for integrating + Django and Dramatiq. +* [django_dramatiq_example][django_dramatiq_example] is a simple + Django app demoing how you can integrate Django and Dramatiq. +* [flask_dramatiq_example][flask_dramatiq_example] is a simple app + demoing how you can integrate Flask and Dramatiq. + + +[taskqueues]: /task-queues.html +[dramatiq]: https://dramatiq.io +[dramatiq-source]: https://github.com/Bogdanp/dramatiq +[django_dramatiq]: https://github.com/Bogdanp/django_dramatiq +[django_dramatiq_example]: https://github.com/Bogdanp/django_dramatiq_example +[flask_dramatiq_example]: https://github.com/Bogdanp/flask_dramatiq_example diff --git a/theme/templates/choices/dramatiq.html b/theme/templates/choices/dramatiq.html new file mode 100644 index 000000000..6e0c9145b --- /dev/null +++ b/theme/templates/choices/dramatiq.html @@ -0,0 +1,18 @@ +

Do you want to learn more about task queues, or another topic?

+
+
+
+ {% include "choices/buttons/task-queues.html" %} +
+
+
+
+ {% include "choices/buttons/deployment.html" %} +
+
+
+
+ {% include "choices/buttons/logging.html" %} +
+
+
diff --git a/theme/templates/table-of-contents.html b/theme/templates/table-of-contents.html index 49d447641..8674f9c73 100644 --- a/theme/templates/table-of-contents.html +++ b/theme/templates/table-of-contents.html @@ -178,6 +178,7 @@

7.5 JavaScript

7.6 Task queues

Celery
Redis Queue (RQ)
+
Dramatiq
Huey

7.7 Static site generators