[Хд] logo

Scaling Python / Django applications

For more than 20 years of existence, Python got a crowd of fans, a large number of modules for all popular platforms and a variety of frameworks. Among the latter a well-deserved popularity of Django, which is used in Instagram, Disqus, Mozilla, Pinterest. Their experience shows that Django handles the highest load and is suitable for scaling projects. Django app

Where to start?

You should begin with profiling, analysis of load on the server and check how the Web application handles the load. Also check out the principles of construction of highload applications and do not forget to implement client optimization.

Caching

After all of the above you can begin finalizing the Django-app. Start with a database cache. Django cache

Django has its own framework for caching. You should create a separate database table for the cache:

python manage.py createcachetable cache_table

# Creates a cache_table

Then it is necessary to supplement the configuration file settings.py with following:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
        'LOCATION': 'cache_table',
    }
}

# You can connect Memcached module for Python in the same way

Now it is possible to cache all the necessary data:

from django.views.decorators.cache import cache_page

@cache_page(60 * 60)
def product_detail(request, product_id):
    ...

# Page cache for 60 minutes (60 seconds * 60 minutes)

Django caching system uses the full address of the request ( "https://www.example.com/stories/2005/?order_by=author") to create a key. But if you want to serve a different content depending on the cookies, language, user agent, the framework can use Vary header to alter output:

from django.views.decorators.vary import vary_on_headers

@vary_on_headers('User-Agent', 'Cookie')
def my_view(request):
    ...

# The system creates a separate caches for different users

Do not forget that you can cache pages with dynamic content. Use tricks. New comments are not necessarily shown at the time of their publication, a few minutes delay is not critical. Note that even 10 seconds caching reduces the load on the entire system. To improve overall performance, you can use the caching with ESI, Varnish, or Memcache.

Static content management

In most cases, it is better not to use Django for static content distribution (except for authorization). But the framework is useful for controlling static files, which can then be served by the web server.

First, you need to edit the configuration file settings.py:

STATIC_ROOT = os.path.join(BASE_DIR, "static")

# It specifies that static files will be stored in the main Web application directory

After you can run static collector:

python manage.py collectstatic

# Collects static files in the specified directory

When Python completes the process, static catalog will appear with all files (CSS, JS, images, etc.) in the subdirectories.

Specify directories with the files:

STATICFILES_DIRS = (
    os.path.join(BASE_DIR, "static"),
)

# Specify catalog with static in settings.py

Better yet, specify the path in the settings of the web server that will do the job faster than Django. For [thehighload.com/index.php/tag/nginx Nginx] you need to add:

server {
    ...
    location /static/ {
        alias /srv/testapp/static/;
    }
}

# These catalogs may be several for different categories of applications

uWSGI

WSGI is Python and Web server communication protocol. And uWSGI is a modern implementation of the protocol. That is, a user sends a request to the web server, which in turn passes it through uWSGI to the Django application, and receives a response. Django uWSGI

With uWSGI you can limit the number of running application processes. It usually executes one parent and two workers. You can alter their number if necessary.

Setting up

uWSGI allow the Django application to communicate with Nginx. First install it:

sudo aptitude install uswgi

# Installation through system packages

The module can also be installed through pip or build from source.

After installation, there will be two new directories: /etc/uwsgi/apps-available and /etc/uwsgi/apps-enabled. The work scheme is simple. It is necessary to put fconfiguration file in apps-available and symlink to it in the apps-enabled.

The testapp.ini file will look like:

[uwsgi]
touch-reload = /tmp/testapp
socket = 127.0.0.1:9001
workers = 2
chdir = /srv/testapp
env = DJANGO_SETTINGS_MODULE=testapp.settings
module = django.core.handlers.wsgi:WSGIHander()

# Specifies the socket number and the workers

Just create an empty file specified in touch-reload:

touch /tmp/testapp

# uWSGI restarts when a file changes

And connect projects:

sudo ln -s /etc/uwsgi/apps-available/testapp.ini /etc/uwsgi/apps-enabled/testapp.ini

# Don’t forget to specify your own path

Now you need to restart uWSGI and configure a web server.

Nginx configuration

The web server needs to be rebuild with ngx_http_uwsgi_module. Then you need to create a configuration file for the Web application /etc/nginx/sites-available/testapp, which will be similar to:

server {
        listen          80;
        server_name     $hostname;
        access_log /srv/testapp/logs/access.log;
        error_log /srv/testapp/logs/error.log;

        location / {
            uwsgi_pass      127.0.0.1:9001;
            include         uwsgi_params;
            uwsgi_param     UWSGI_SCHEME $scheme;
            
        }

        location /static {
            alias /srv/testapp/static/;
            index  index.html index.htm;

        }
}

# With the indication of the socket and addresses to static files

You can also configure the cluster using Nginx as a balancer:

upstream uwsgicluster {
     server 127.0.0.1:9001;
     server 192.168.100.101:9001;
     server 192.168.100.102:9001;
     server 192.168.100.103:9001;
     server 192.168.100.104:9001;
}

# Nginx will distribute the load evenly

The most important

Caching and the uWSGI module can increase the performance and speed of Django Web application. This architecture scales easily and does not require additional investments. Do not forget to optimize your web server configuration to improve the responsiveness of the site.

  читать на русском
[Хд]

Sign up to read high quality stuff on advanced development

Google Email

Esc for later