Django: works with uwsgi, or even works with uwsgi & nginx together

Foundations

Django: Python web framework

uWSGI: Application server, handles Python code execution

Nginx: Web server, handles static files and reverse proxy

The final architecture of this page will be:

Browser -> Nginx -> uWSGI -> Django

Basic Environment

I'm using venv + poetry here:

venv for virtual environment management

poetry to enhance venv: venv doesn't manage dependency packages
1. I used to frequently use requirements.txt with pip, but I’m tired of it.

Setting up the Environment

# Create virtual environment
python -m venv myenv

# Activate virtual environment
source myenv/bin/activate

# Use Poetry for dependency management
python -m pip install poetry

Package Management

It's kind of like when you do npm init

I've written golang before and you can use go tidy to automatically register needed packages and delete unnecessary ones. Too bad poetry doesn't have this feature, you can only add slowly.

# manually add
poetry init  # create pyproject.toml
poetry add django
poetry add uwsgi

poetry install will refer to existing file

poetry.lock

pyproject.toml (if no poetry.lock)

If not, you manually poetry add ...

But when pulling a project from remote, you usually have .lock or .toml files, you can directly poetry install

Django Basic Setup

The most basic Django installation, start a project, make a ping in that project

django-admin startproject mywebsite
cd jialinwebsite

File structure

jango/
├── manage.py
├── mywebsite/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py

Of course, you'll want to add other routes:

django-admin startapp ping

File structure

mywebsite/
├── manage.py
├── mywebsite/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── ping/
    ├── __init__.py
    ├── views.py
    ├── urls.py
    └── apps.py

In ping/views.py

from django.http import HttpResponse

def ping_response(request):
    return HttpResponse("pong")

In ping/urls.py

from django.urls import path
from . import views

urlpatterns = [
    path('ping/', views.ping_response, name='ping_view'),
]

In the main mywebsite/urls.py

from django.contrib import admin
from django.urls import include, path
urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('ping.urls')) # add this
]

A Quick Note

In Django, naming URL patterns (like name='ping_view' from urls.py) has several benefits:

Use in templates

<a href="{% url 'ping_view' %}">Ping Page</a>

Or use in views

from django.urls import reverse
from django.http import HttpResponseRedirect

def some_view(request):
    return HttpResponseRedirect(reverse('ping_view'))

If you need to change the URL structure in the future (e.g., from ping/ to greetings/), you only need to modify urls.py, without updating all references in templates and views.

Run!

python manage.py runserver

👏 Now you can start a simple django!

Adding uWSGI

Why do we need uWSGI?

uWSGI as an application server can handle Python code execution more effectively and work well with web servers like Nginx.

The original thin django is convenient for development, but when it comes to actually becoming a publish, it will have problems handling high traffic, and although it's a multi-thread framework, it needs to rely on external tools to achieve this.

Although it can handle application-level errors like 404, 500, it can't handle server-level errors, and doesn't have built-in SSL settings, only relying on external integration.

uWSGI as an application server can handle Python code execution more effectively and can work well with web servers like Nginx later.

Basic Configuration

pip install uwsgi

# uwsgi.ini
[uwsgi] # must have this section
http = :9555
module = mywebsite.wsgi:application
vacuum = True # do clean up after shut down
pidfile = /tmp/wsgi.pid # if you want tracking this process.

Start django with uwsgi, and you can see the django server at localhost:9555.

uwsgi --ini uwsgi.ini

👏 Now at this point, you've integrated uwsgi + django, next we'll integrate uwsgi+nginx.

Here comes nginx

Before

uWSGI directly serves as an HTTP server, handling HTTP requests from clients

Browser -> uWSGI -> django

django+uwsgi only

Now

But now you might consider: nginx is better at handling security like SSL, and better performance on static files, and if you have multiple uWSGIs, you can also let nginx proxy multiple uWSGIs to achieve load balancing.
Now I want to change uWSGI's responsibilities, uWSGI no longer directly handles HTTP requests, but communicates with Nginx through a socket.

Browser -> nginx -> Unix.socket -> uWSGI -> django

`uwsgi.ini` now uses socket communication instead of http

Now we're integrating nginx, so of course, the communication method needs to change. The socket field below is when uwsgi --ini uwsgi.ini starts running, it will create a socket to communicate with nginx.

[uwsgi]
module = jialinwebsite.wsgi:application
# %(base_dir)s/mywebsite.sock
socket = /Users/jialinhuang/Desktop/jango/jialinwebsite/www.sock
vacuum = True
pidfile = /tmp/wsgi.pid
debug = True

Install nginx

brew install nginx

Configure nginx related files

Create custom configuration folder, and add files for the next step.

Also nginx files on macOS are in /opt/homebrew/etc/nginx/

mkdir -p /opt/homebrew/etc/nginx/sites-available
touch /opt/homebrew/etc/nginx/sites-available/mywebsite
vim /opt/homebrew/etc/nginx/sites-available/mywebsite

Create site configuration file, this is the nginx conf file we'll use later

# /opt/homebrew/etc/nginx/sites-available/mywebsite
server {
    listen 80;
    server_name localhost;

    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /path/to/your/static/files;
    }

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/Users/jialinhuang/Desktop/jango/jialinwebsite/www.sock;
    }
}

Create soft link to sites-enabled
The second step is you as a candidate, this step is you as an elected one
```
sudo ln -s /opt/homebrew/etc/nginx/sites-available/mywebsite /opt/homebrew/etc/nginx/sites-enabled/
```
/nginx/sites-available vs. /nginx/sites-enabled
Both store config files, but sites-available can be considered candidates, and sites-enabled are actively enabled.

Include custom configuration in the main Nginx configuration file
Add this line include /opt/homebrew/etc/nginx/sites-enabled/*;
to
/opt/homebrew/etc/nginx/nginx.conf in the http block
Why? Because every time nginx restarts, it will include that it needs to include the configuration files in site-enabled.
```
# /opt/homebrew/etc/nginx/nginx.conf
http {
    include       mime.types;
    default_type  application/octet-stream;
    include /opt/homebrew/etc/nginx/sites-enabled/*;
}
```

nginx user configuration

The reason will be explained below

user <username> staff;
# or
user root staff;

nginx restart

sudo nginx -t  # Test configuration
sudo nginx -s reload  # Reload configuration

Finally, remember to start uwsgi.... Because nginx is just doing reverse proxy, remember to start your uwsgi service behind it

Note

Make sure the uwsgi_pass path in your custom /opt/homebrew/etc/nginx/sites-available/jialinwebsite file matches the socket path in uwsgi.ini

Confirm the static file path is correct.

Make sure the Nginx main configuration file can load configurations in sites-enabled, and your own configuration should be in site-enabled/.

And whenever there's any change in nginx configuration, remember sudo nginx -s reload, always.

DEBUG Nginx unable to connect to uWSGI socket

This is very important and related to the user field at the very beginning of /opt/homebrew/etc/nginx/nginx.conf

After I set it up, I found that nginx couldn't connect no matter what, and there was no message on the uwsgi backend, so I checked

tail -f /opt/homebrew/var/log/nginx/error.log

fail log

2024/10/02 23:55:20 [crit] 22768#0: *1 connect() to 
unix:/Users/jialinhuang/Desktop/jango/jialinwebsite/www.sock 
failed (13: Permission denied) while connecting to upstream, 
client: 127.0.0.1, server: localhost, 
request: "GET /ping HTTP/1.1", 
upstream: "uwsgi://unix:/Users/jialinhuang/Desktop/jango/jialinwebsite/www.sock:", 
host: "localhost”

It should be insufficient permissions to operate www.sock, so I checked the permissions of this file

 $  ls -l jialinwebsite                                   on  main!
-rw-r--r--   1 jialinhuang  staff  583  3 Oct 10:53 uwsgi.ini
srwxr-xr-x   1 jialinhuang  staff    0  3 Oct 10:53 www.sock

TRY 1: Set chmod-socket = 777 to deliberately make the socket in the most freely accessible state?

Set chmod-socket = 777 in uwsgi.ini, and /opt/homebrew/etc/nginx/nginx.conf is unchanged by default, which means it doesn't have a user field

socket = /Users/jialinhuang/Desktop/jango/jialinwebsite/www.sock
chmod-socket = 777

/opt/homebrew/etc/nginx/nginx.conf

#user  nobody;

It doesn't work. When Nginx doesn't specify a user to run as, it might run as the system's default low-privilege user (like nobody or www-data). These low-privilege users might not be able to enter the directory where the .sock file is located, even if the file itself has 777 permissions, the directory's permissions also need to allow access.

TRY 2: Specify the user field, and don't give chmod-socket = 777 in uwsgi.ini

socket = /Users/jialinhuang/Desktop/jango/jialinwebsite/www.sock
# chmod-socket = 777

No need to manage chmod, but just write this in /opt/homebrew/etc/nginx/nginx.conf

# user  nobody;

user jialinhuang staff; # yes, it works
user root staff;        # yes, it works
user jialinhuang;       # no, do not use
user root               # no, do not use

So from the above configuration, we can know that it's not related to chmod, but more related to what identity nginx runs as.

Socket Type Comparison

TCP socket vs. Unix socket

We mentioned earlier that the uwsgi_pass in /opt/homebrew/etc/nginx/sites-available/jialinwebsite and the socket field in uwsgi.ini must be the same.

You can choose to use Unix Socket, which is for local use and can avoid network overhead, so it's limited to inter-process communication(IPC) on the same machine. At the beginning above, we used the Unix method, directly writing the file path.

Unix socket

# /opt/homebrew/etc/nginx/sites-available/jialinwebsite
uwsgi_pass unix:/path/to/your/mywebsite.sock;

# uwsgi.ini
socket = /path/to/your/mywebsite.sock

If you want to change to TCP socket, you can, after all, in addition to many network testing tools, it can also make cross-machine possible. The following is simply changing the original unix to TCP

# /opt/homebrew/etc/nginx/sites-available/jialinwebsite
server {
    # ...
    location / {
        include uwsgi_params;
        uwsgi_pass 127.0.0.1:8000;
    }
}

[uwsgi]
socket = 127.0.0.1:8000

Why is TCP Socket better when thinking about load balancing?

Look, below is the cross-machine writing method.

When I get to the root / nginx will know to give it to backend, backend has two machines that can be sent to, you can also set weight to achieve traffic ratio adjustment

# if is 
# Unix socket (same host)
upstream backend {
    server unix:/path/to/socket1.sock;
    server unix:/path/to/socket2.sock;
}

# else 
# TCP socket (different hosts)
upstream backend {
    server 192.168.1.10:8000;
    server 192.168.1.11:8000;
}

# load balancing with TCP socket
upstream backend {
    server 192.168.1.10:8000 weight=3;
    server 192.168.1.11:8000;
}

Is pipx design against its original intention? (You can skip this part)

I defaulted to the idea of node's npx, thinking I'd use pipx to run python first, but found many inconveniences, not as intuitive to use. For example:

I thought I could directly use python when I entered, but apparently python and pipx environments are different. Essentially, pipx is a global virtual environment, not globally available. Even if you're not in a virtual environment yourself, pipx is its own set of virtual environments?

The following is my imitation of the initial setup, globally installing django CLI with pipx for convenience later. Using django alone is no problem, just slightly different from the commands set at the beginning of this article, and it runs

# go to directory
cd pipx-django
pipx install django

# create project
django-admin startproject mywebsite
cd mywebsite
django-admin startapp ping

# do some view.py, urls.py .. setup


# RUN
django-admin runserver

But when I wanted to run through uwsgi.ini

uwsgi --ini uwsgi.ini

error log

[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.27 (64bit) on [Thu Oct  3 18:18:52 2024] ***
compiled with version: Apple LLVM 15.0.0 (clang-1500.3.9.4) on 02 October 2024 14:21:09
os: Darwin-23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:43 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6000
nodename: JiaLins-MacBook-Pro.local
machine: arm64
clock source: unix
pcre jit disabled
detected number of CPU cores: 10
current working directory: /Users/jialinhuang/Desktop/pipx-django/mywebsite
writing pidfile to /tmp/wsgi.pid # if you want tracking this process.
detected binary path: /Users/jialinhuang/.local/pipx/venvs/uwsgi/bin/uwsgi
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 2666
your memory page size is 16384 bytes
detected max file descriptor number: 256
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :9555 fd 4
spawned uWSGI http 1 (pid: 86592)
uwsgi socket 0 bound to TCP address 127.0.0.1:63803 (port auto-assigned) fd 3
Python version: 3.12.6 (main, Sep  6 2024, 19:03:47) [Clang 15.0.0 (clang-1500.3.9.4)]
Python main interpreter initialized at 0x101880e00
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72896 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
Traceback (most recent call last):
  File "/Users/jialinhuang/Desktop/pipx-django/mywebsite/mywebsite/wsgi.py", line 12, in <module>
    from django.core.wsgi import get_wsgi_application
ModuleNotFoundError: No module named 'django'
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 86591, cores: 1)

Simply put, it can't find Django. This is because when uWSGI starts up, it uses the system Python path to look for installed packages. Django is managed by pipx and located in a global virtual environment, so uWSGI can't automatically find Django.

That's why uwsgi needs this extra configuration line to explicitly specify where my virtual environment is: virtualenv = /Users/jialinhuang/.local/pipx/venvs/django

# uwsgi.ini
[uwsgi]
http = :9555
module = mywebsite.wsgi:application
vacuum = True
pidfile = /tmp/wsgi.pid
virtualenv = /Users/jialinhuang/.local/pipx/venvs/django

This really goes against pipx's original intention, doesn't it?

Sure, pipx simplifies a lot of global settings, but it adds some complexity in actual development.

At first, I naively thought it would be intuitive like my nodejs npx concept and just used it, but I found pipx isn't that intuitive. And the issues it's trying to solve still end up happening anyway.

Uh. Bye

brew uninstall pipx
rm -rf ~/.local/pipx

References

https://unix.stackexchange.com/questions/111346/what-is-the-difference-between-gid-and-groups

http://nginx.org/en/docs/beginners_guide.html