My Python development template in 2022
I wish I had this when I started coding! A template for containerized Python development with packaging, CI/CD, code formatting, testing and more.
Inspiration
I recently came across a great article by Mathspp called How to create a Python package in 2022. It provides a comprehensive guide to creating a Python package and it includes some nice extras like dependency management with Poetry, CI/CD setup, pre-commit hooks, testing with pytest, automation using tox, etc.
The post by Mathspp inspired me to formally document my current setup, firstly to solidify it for myself, but also to share with others. This type of template is something that I wish I had when I started my development journey years ago.
A lot of the tools that Mathspp uses aligns with my own current setup but there are also some differences. The main difference in my setup is Docker so let me show you why I like it!
Using Docker
I've mainly used Miniconda for package, dependency and environment management in recent years, mainly for data science and machine learning projects. The simplicity and ease of use of Conda makes it easy to start up quickly and install packages without polluting your local environment, but it lacks in some areas, specifically when it comes to production.
Conda environments are often bloated with unnecessary dependencies, making them larger than needed, and they are reliant your local machine's configuration.
Portability for production
When it comes to production, Docker is king because of its portability and reproducibility at scale. It is easy to replicate your exact environment at scale and most cloud providers have built a lot of their infrastructure around Docker systems.
VS Code devcontainer
VS Code makes developing inside a Container easy, allowing you to create a Docker container and open any folder inside (or mounted into) the container. The benefits include a highly portable and reproducible development environment and the Docker container can often be used for CI/CD and production with minor modifications. This speeds up the process of taking a project from development into production with CI/CD.
Easy experimentation
Say you need to test a piece of code on a new OS or Architecture, simply head over to Docker Hub, select your image of choice (alpine, busybox, python, postgres, redis, traefik, ubuntu, node, golang, etc!), pick the relevant version (tag) and pull away. You can event spin up 'n quick devcontainer using the image if you like. Then, go crazy inside your container by installing stuff, messing around with settings, digging through the filesystem, etc. And nothing touches your local setup.
When you're done, simply throw away the image (or start again if you broke something!). Alternatively, if you want to keep working with an image you can write up your exact process in a Dockerfile and add it to your stack. Simple as that, all very convenient.
CI/CD pipelines
When it comes to CI/CD, it will probably also happen on a runner that spins up a Docker container. You can install obviously still install conda/venv/etc. and build your environment inside docker, or alternatively you can develop inside a container to start with and completely remove the need for a virtual environment. Using Docker makes CI/CD development easier and quicker because you can figure out most process locally first before committing to a CI/CD pipeline.
Python project template
The template can be found on my github page here: https://github.com/jacoverster/python-project-template
Main features
The main features of the template are:
- VS Code .devcontainer configuration that installs commonly used VS Code extensions.
- Dockerfile with Python and the local package with DEV dependencies installed.
- pyproject.toml package configuration with optional DEV and TEST dependency lists.
- Pytest template setup with code coverage reporting.
- Pre-commit hooks configuration with excellent hooks for better code quality and safety.
- Black code formatting.
- GitHub Actions configuration for automated testing and building on GitHub.
- GitLab CI/CD configuration for automated testing and building on GitLub.
- README template from makeareadme.
- LICENSE file.
Updates
I plan to update the template as I go to keep it up to date with my latest setup as far as possible. It also helps to have a static reference for my projects, every project has it's own requirements but I always start from some baseline.