Python Environments (again!)

Here I am again, trying to run multiple Pythons, with multiple libraries, without them tripping over each other. Many hackers have gone down this path and become lost, I expect I will fare no better, but I've stumbled upon a scheme which works for me, whether running Python on a Macintosh, in a Linux server, or even on Windows.

Like everyone, I have been having a love-hate relationship with python virtual environments. I even had a set of bash functions to wrap virtualenv, like virtualenvwrapper. This worked, but had some problems, and while it supported Conda and Virtualenv, it didn't address managing multiple different Python runtimes.

Which Python manager to use, and when?

What I think I have decided today is this:

1. Avoid using “system” pythons for programming projects, including Homebrew on macOS — that counts as a system python.
   • System pythons are for use by utilities and applications which are installed and managed for you by the OS package system.
   • They can and do change unexpectedly (particularly Homebrew is bad at this), which breaks dependent virtual environments, making them unsuitable for doing programming work
   • Leave system python runtimes to your package system and develop with your own python runtimes.
2. Use conda (Anaconda) for setting up anything “complicated” (Scientific stuff, CUDA, Jupyter, using pygame on a Mac, or using python at all on Windows).
   • Conda supports using multiple environments, and it just works.
   • You can use pip in a conda environment, for PyPI packages which aren't provided by Anaconda, or Conda Forge.
   • But for other development projects (or Linux server deployments) it may be easier to use “traditional” light-weight python environments
     (miniconda is lightweight, but more difficult to set up than Anaconda Full, at least in Mac/Windows, and it is less common in servers).
3. Use ASDF to manage multiple pythons (but not venvs) for development and testing.
   • This deserves its own blog post, I have a few issues about this in another project.
   • ASDF uses pyenv under the hood. Pyenv compiles python runtimes, Unix Ports-style.
   • But some versions of python can't be compiled on newer macOS (e.g. Python 3.6).
4. As a last resort (or first option on a server), use Docker Containers, or virtual machines, to host python environments.
   • These run python on the Linux kernel in the container/virtual machine.
   • You can install any version of python, typically as part of the container image, or as a package of the guest OS.
   • It's how I run this blog's site generator, since that needs python 3.6.
   • Manage multiple pythons at the machine/container level, one single environment per machine, which side-steps the multiple pythons issue entirely.
   • But these can be more difficult to manage, and require virtualization hardware and software.
   • And access to the host environment (e.g. to interface with files or games running on the host) is more difficult.

Virtual environments

1. In unix workstations, Pipx is a nice way to install PyPI applications “stand-alone” — for instance bpytop, or ansible. These dedicated environments are lightweight and linked the python which runs pipx, by default (this can be overridden with the --python option).
   • Pipx is installed as a Homebrew package on macOS, and a pip package elsewhere (see my environment variable settings below, first): brew install pipx, or python -m pip install --user pipx
2. For python virtual environments, if you can't use Conda environments (because you are not using Conda), then use virtualenv — it's the next most mature option.
   • I've flip-flopped on this decision a few times in the past, but I have come back into the virtualenv fold for vanilla python, and also using conda to work around virtualenv's limits for Windows and Mac.
   • You can install virtualenv with pipx: pipx install --python $(asdf which python) virtualenv
     • If you installed pipx with Homebrew, then this means that the default python for any virtualenv-managed environment is also Homebrew's python.
     • Since this is changeable on Homebrew's whim, you need to override it with --python $(asdf which python) to use the ASDF global python.
3. Python's built-in venv module is the PEP 405 standard, but it's less capable than virtualenv and slower.
4. I've always disliked virtualenvwrapper, but VirtualFish (despite the unhelpful name) provides the same functionality, with better names, and Tide prompt support.
   • Install virtualfish: pipx install --python $(asdf which python) virtualfish, (to avoid Homebrew as the default python for venvs)
   • Then install it into fish with vf install; alias vp=vf; funcsave vp.
5. Avoid using pipenv. It only offers dependency locking and some convenience (e.g. “automatic” activation via pipenv shell), at the cost of brokenness. These are better provided by other means:
   • pip-tools pinned/locked pip dependencies with the ability to keep them fresh
   • VirtualFish has a plugin for automatic environment activation, or else use direnv, like this.

Configuring default locations for venvs

I have strong opinions on where things belong. Here are some unix environment variables to control where virtualenvs or pipx applications are installed:

#virtualenv / virtualfish base directory
#https://pipenv.pypa.io/en/latest/advanced/#custom-virtual-environment-location
WORKON_HOME=~/lib/venv
#https://github.com/justinmayer/virtualfish/blob/main/virtualfish/virtual.fish
VIRTUALFISH_HOME=~/lib/venv

#PIPX https://pypa.github.io/pipx/docs/
PIPX_HOME=~/lib/pipx
PIPX_BIN_DIR=~/bin