PPTX, PDF749 views

PyGrunn 2017 - Django Performance Unchained - slides

This document discusses optimizing Django performance. It begins by describing the server, Apache, database, and Django configurations. It then discusses optimizing various aspects of the Django project like the ORM, database queries, template rendering, caching, and DjangoCMS-specific optimizations. Key recommendations include using select_related, prefetch_related, batch database operations to avoid queries in loops, optimizing templates, reducing cache misses, and handling {% page_url %} tags efficiently in DjangoCMS projects. The document provides examples of inefficient code and more optimized alternatives.

Technology◦

Bottlenecks
● Server
● Apache
● Database
● Django
● Sessions
● AJAX API
● Search
● iSeries hammering

Server specs
● Virtualized environment
● Single VPS, horizontal scaling possible
● Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
● 24 cores
● 96GB RAM
● RHEL7

Apache 2.4
● Prefork MPM has a high overhead
● Event MPM scales really well
<IfModule event.c>
StartServers 10
MinSpareThreads 160
MaxSpareThreads 800
ThreadLimit 80
ThreadsPerChild 80
ServerLimit 20
MaxRequestWorkers 1280
MaxRequestsPerChild 0
</IfModule>

Database
● PostgresQL
● Tuning:
max_connections 100 1000
shared_buffers 32MB 8GB
work_mem 1MB
64MB
maintenance_work_mem 16MB 256MB
effective_cache_size 128MB 2GB
log_min_duration_statement -1 500
log_checkpoints off on
log_connections off on
log_disconnections off on
log_line_prefix '' '%d %s
%m: '
log_lock_waits off on

Django
● Running under Gunicorn
● 32 workers

Sessions
● In most cases, we don’t need sessions
● When we do, memcached

Search
● Static parameters
● Volatile availability
● Set operations
● Facet counting
● Why not elasticsearch

Halfway Summary
● Full-stack approach:
● Tuning of server, database, apache
● Varnish reverse proxy
● API desing
● Smart search filtering/faceting with set operations
● Caching of iSeries calls (Python)

Django optimisations
What costs time (and what can we optimise)?
● ORM
● Database queries
● Abusive cache usage
● Template rendering

django-debug-toolbar
https://pypi.python.org/pypi/django-debug-toolbar

django-debug-toolbar-template-profiler
https://pypi.python.org/pypi/django-debug-toolbar-template-profiler

Before we move on
django-debug-toolbar is an excellent tool, but page rendering timings lie.

Imaginary app (page 2)
class Book(models.Model):
title = models.CharField(max_length=100)
authors = models.ManyToManyField('books.Author', related_name='books')
publisher = models.ForeignKey(Publisher, related_name='books')
publication_date = models.DateField()
isbn = models.CharField(max_length=100, unique=True)
price = models.DecimalField(max_digits=10, decimal_places=2)
pages = models.PositiveIntegerField(default=200)
stock_count = models.PositiveIntegerField(default=30)

Imaginary app (page 3)
class Order(models.Model):
owner = models.ForeignKey(settings.AUTH_USER_MODEL)
lines = models.ManyToManyField("books.OrderLine", blank=True)
created = models.DateField(auto_now_add=True)
updated = models.DateField(auto_now=True)
class OrderLine(models.Model):
book = models.ForeignKey('books.Book', related_name='order_lines')

$ORM & Database queries Tips: ● Make use of `select_related` and `prefetch_related` for fetching related objects at once. ● Make use of ònly` and `defer` (typical use case: listing views), but use them with caution. ● Use ànnotate` and àggregate` for calculations on database level. ● When fetching IDs of related item (if you only need ID), use {field_name}_id instead of {field_name}.id, since the first one does not cause table JOIN. ● Consider using `values` or `values_list` for listing views, when possible.$

Use case
Listing of books (about 2000 in total) with the following information:
● Book title
● Names of all authors of the book separated by comma
● Publisher name
● Number of pages
● Price

Results
4023 queries took 730 ms, page rendered in 23932 ms

Let’s improve
Adding:
● select_related
● prefetch_related
● only

Good example
books = Book.objects.all()
.select_related('publisher')
.prefetch_related('authors')
.only('id', 'title', 'pages', 'price',
'publisher__id', 'publisher__name',
'authors__id', 'authors__name')

Results
2 queries took 12 ms, page rendered in 2104 ms

Let’s improve more
Adding:
● values
● annotate

Even better example
books = Book.objects.all()
.values('id', 'title', 'pages', 'price',
'publisher__id', 'publisher__name')
.annotate(
authors__name=GroupConcat('authors__name', separator=', ')
)
.distinct()

Results
1 query took 13 ms, page rendered in 568 ms

But what is GroupConcat?
But what is GroupConcat?

Custom aggregation functions
from django.db.models import Aggregate
class GroupConcat(Aggregate):
function = 'group_concat'
@property
def template(self):
separator = self.extra.get('separator')
if separator:
return '%(function)s(%(field)s, "%(separator)s")'
else:
return '%(function)s(%(field)s)'

Avoid database hits in the loop!
Operate in batch as much as possible!

Bad example
for __i in range(1, 50):
author = Author.objects.create(
salutation='Something %s' % uuid.uuid4(),
name='Some name %s' % uuid.uuid4(),
email='name%s@example.com' % uuid.uuid4()
)

Results
98 queries took 40 ms, page rendered in 652 ms

Good example
authors_list = []
for __i in range(1, 50):
author = Author(
salutation='Something %s' % uuid.uuid4(),
name='Some name %s' % uuid.uuid4(),
email='name%s@example.com' % uuid.uuid4()
)
authors_list.append(author)
Author.objects.bulk_create(authors_list)

Results
2 queries took 1.73 ms, page rendered in 60 ms

If you use django-debug-toolbar already...
...you may say

I know it, django-debug-toolbar is great...
...but what about non-HTML (JSON, partial HTML) views?

django-debug-toolbar-force
https://pypi.python.org/pypi/django-debug-toolbar-force
MIDDLEWARE_CLASSES += (
'debug_toolbar.middleware.DebugToolbarMiddleware',
'debug_toolbar_force.middleware.ForceDebugToolbarMiddleware',
)
And add ?debug-toolbar to any non-HTML (JSON, AJAX) view URL

One important thing!
And you’ll hear it many times
in this presentation...

Given the following model...
class Item(models.Model):
text = models.TextField()
page_id = models.IntegerField()
...where page_id is the correspondent ID
of the CMS page

Because it happens and you have to deal with it!

Bad example
items = Item.objects.all()
for item in items:
page = Page.objects.get(id=item.page_id)
# ... do something else with item

$Improved example # Freeze the queryset items = Item.objects.all()[:] # Collect all page ids page_ids = [item.page_id for item in items] pages = Page.objects.filter(id__in=page_ids) pages_dict = {page.id: page for page in pages} for item in items: page = pages_dict.get(item.page_id) # ... do something else with item$

Results
No additional database queries and no missed cache hits.

$Unnecessary cache hits Tips: ● Analyze cache usage with django-debug-toolbar. ● Use template fragment caching to minimize the number of cache queries. ● Identify heavy parts of your templates with TemplateProfilerPanel. Optimise them first and after that, if they are still heavy, cache them. ● Try to avoid repetitive missed cache queries. ● The well known {% page_url %} and {% page_attribute %} tags of DjangoCMS may produce a lot of missed cache queries. ● Document cache usages. Explain in the code why do you cache that certain part. Keep track of all cache usages in a separate section of your developer documentation.$

$Optimise templates Tips: ● Clean up your templates. Remove all unused import statements. ● Use cached template loading on production. ● Avoid database queries done in the loop. Especially {% page_url %} and {% page_attribute %} template tags of DjangoCMS.$

A couple of things
● Try to reduce the number of database queries
● Use EXPLAIN when things are still slow.
● Add indexes when necessary.
● Compare the "before" and "after". Do it often.
● Test page load speed with caching on and off.
● Test the first page load and the second page load. Analyze the difference.
● Try to reduce the number of missed cache queries.

Tips
● Try to get rid of context processors and middleware that hit the database.
● Be careful with templatetags that query the database/cache. Especially,
when they are done in a loop.

DjangoCMS is fine...
...but all your non-CMS views would be affected with
additional checks and queries, even if they are totally
irrelevant.

Hold on, there’s a way to fix it!
# Some sort of a fake page container, to trick django-cms.
PageData = namedtuple('PageData', ['pk'])
class MyView(View):
def get(self, request):
# This is to trick the django-cms middleware, so that we don't
# fetch page from the request (not needed here).
request._current_page_cache = PageData("_")
context = self.get_context(request)
return render_to_response(
self.template_name),
context,
request
)

$Dealing with {% page_url %} Avoid: ● Use of {% page_url %} or {% page_attribute %} tags on IDs. Pass complete objects only. ● Apply tricks to the querysets to fetch the page objects efficiently. ● If you list a lot of DjangoCMS Page objects (with URLs) on a page, don’t use {% page_url %} at all. Instead, use customized template tag which fits your needs better.$

Use case
Sitemap page with about 200 CMS pages.

$Bad example pages = Page.objects.all() {% for page in pages %} {% page_url page.id %} {% endfor %}$

Results
2000 database queries and 3000 missed cached hits

Improved example (page 1)
pages = Page.objects.all()[]
page_ids = [page.id for page in pages]
titles = Title.objects
.filter(language=get_language())
.filter(page__id__in=page_ids)
.select_related('page', 'page__publisher_public', 'page__site')
.only('id', 'title', 'slug', 'path', 'page__id', 'page__reverse_id',
'page__publisher_is_draft',
'page__publisher_public__publisher_is_draft',
'page__publication_date', 'page__publication_end_date',
'page__site', 'page__is_home', 'page__revision_id')[:]

$Improved example (page 2) pages_dict = {} for _title in titles: if not hasattr(_title.page, 'title_cache'): _title.page.title_cache = {} _title.page.title_cache[get_language()] = _title pages_dict[_title.page.id] = _title.page _pages = [] for page in pages: _pages.append(page_dict.get(page.id, page)) pages = _pages$

Improved example (page 3)
class AddonsPageUrlNoCache(PageUrl):
"""PageUrl made that doesn't hit the cache.
You should not be using this everywhere, however if you know what you're
doing, it can save you a lot of page rendering time.
"""
name = 'addons_page_url_no_cache'

Improved example (page 4)
def get_value(self, context, page_lookup, lang, site):
site_id = get_site_id(site)
request = context.get('request', False)
if not request:
return ''
if lang is None:
lang = get_language_from_request(request)
page = _get_page_by_untyped_arg(page_lookup, request, site_id)
if page:
url = page.get_absolute_url(language=lang)
if url:
return url
return ''

$Improved example (page 5) register.tag(AddonsPageUrlNoCache) {% for page in pages %} {% addons_page_url_no_cache page %} {% endfor %}$

Use get_render_queryset on CMS plugins
Use get_render_queryset class method to fetch objects efficiently.

Use case
Write a CMS plugin which has 2 foreign key relations.

Bad example
class TextPictureLink(AbstractText):
title = models.TextField(_("Title"), null=True, blank=True)
image = FilerImageField(null=True, blank=True)
page_link = PageField(null=True, blank=True)
class GenericContentPlugin(TextPlugin):
module = _('Blocks')
render_template = 'path/to/generic_text_plugin.html'
name = _('Generic Plugin')
model = models.TextPictureLink

Results
3 database hits for a single plugin. How many plugins do you have on a page?

Good example
class GenericContentPlugin(TextPlugin):
# ... original code
@classmethod
def get_render_queryset(cls):
"""Tweak the queryset."""
return cls.model._default_manager
.all()
.select_related('image', 'page_link')

Results
1 database hit for a single plugin.

Apache JMeter
Highlights:
● Load/stress testing tool for analyzing/measuring the performance.
● Focused on web applications.
● Plugin based architecture.
● Supports variable parametrisation, assertions (response validation), per-
thread cookies, configuration variables.
● Comes with a large variety of reports (aggregations, graphs).
● Can replay access logs, including Apache2, Nginx, Tomcat and more.
● You can save your tests and repeat them later.