How we built a clustered solution for dozens of small Django/Python applications
For a long time, we have been solving the question of how to host our projects in Python/Django so that it is easy to maintain, secure and of course stable. We started with a classic VPS, on which all dependencies, databases, web servers, and just everything the application needs to run were installed. As the projects grew, we scaled the VPS vertically. Over time, we found that one VPS was insufficient, so we added two more instances (a test environment and a production environment). The new VPSs were already being installed using ansible scripts to provide exactly the same test environment as the production one. In addition to our own VPS, we had several other VPS clients under management running our applications, running different OSes (Ubuntu, Debian, Centos) and of different ages. All of this accumulated, resulting in a lot of inconsistency in the individual solutions and a challenge to manage and maintain.
Problems of VPS hostings:
- Different versions of the OS have different versions of the packages in the repositories. Automatic scripts may not work the same way everywhere (ansible partially solves this problem, but not 100%).
- Using different server applications (Apache, Nginx, MySQL, Postgres) requires different configurations, backup scripts, etc.
- VPS has limited storage. You can add or expand disks, but it's always a lot of manual work you don't want to do.
- A server outage will cause all projects on that server to fail.
- You are afraid to upgrade the VPS lest something goes wrong and upgrading to a higher version is a nightmare.
- You can only scale the VPS vertically. Upgrading other VPSs brings worries about other systems.
- We deploy differently on different VPSs (semi-automatically using fabfile, ansible).
- Each project brings with it more and more dependencies that need to be installed on the server and which may even conflict with each other in the end.
The above problems led us to change the hosting to one universal, scalable, long-term sustainable, and stable solution in the form of a Docker Swarm cluster.
Current configuration of our cluster solution
For hosting we still use VPS servers (4) over which is built Docker Swarm, which wraps all VPS into one cluster in which individual applications can run on any VPS without limitations. Due to the fact that applications can migrate between VPSes, it is necessary to switch to "stateless" applications (all persistent data must be stored outside the application/VPS). Because of this requirement and also for ease of management, we have switched to as many managed services as possible (PostgreSQL, MySQL, Redis, S3 storage, Sendgrid SMTP).
The gateway to the cluster is Traefik (webserver), which knows about all applications within the cluster and acts as a reverse proxy for the application servers. The individual nodes then run Docker containers and the applications in them (gunicorn application server, Django).
Among other things, Traefik solves:
- routing HTTP requests within the cluster
- SSL certification
- redirects
- HTTP headers
Traefik learns about the application using labels that are part of the deployment config for Docker Swarm within each application. The advantage is that these config files reside with the application source code, so you have everything about the application in one place.
labels:
- traefik.http.routers.vistacredit-web-develop-http.entrypoints=http
- traefik.http.services.vistacredit-web-develop.loadbalancer.server.port=80
- traefik.http.routers.vistacredit-web-develop-http.middlewares=non-www-redirect-secure
- traefik.http.routers.vistacredit-web-develop-https.middlewares=non-www-redirect-secure
- traefik.http.routers.vistacredit-web-develop-https.entrypoints=https
- traefik.http.routers.vistacredit-web-develop-https.tls=true
- traefik.http.routers.vistacredit-web-develop-https.tls.certresolver=le
- traefik.http.routers.vistacredit-web-develop-http.rule=Host(`vista.doc.endevel.cz`)
- traefik.http.routers.vistacredit-web-develop-https.rule=Host(`vista.doc.endevel.cz`)Benefits of the current solution:
- Easy horizontal scaling by adding a new Node and calling a single command (docker swarm init).
- When one VPS (node) fails, automatically moves containers to another node.
- Unlimited storage (thanks to the use of S3)
- Managed DB (you really don't want to worry about the DB even if you think you don't need to).
- Automatic deployment using GitLab Pipelines.
- Dockerized environment for applications.
- Exactly the same testing environment as production.
- Easily add other environments (stage, dev) just by adding a deployment config.
- Easy updates and upgrades to individual VPS.
- On-demand test applications (applications only run on demand - when requested).