|
| 1 | +--- |
| 2 | +title: 'Venv at home: My solution to "error: externally-managed-environment"' |
| 3 | +tags: python |
| 4 | +redirect_from: /p/79 |
| 5 | +--- |
| 6 | + |
| 7 | +Since Python 3.12 (which enforces [PEP 668](https://peps.python.org/pep-0668/)) shipped with Ubuntu 24.04, like many others, I found my usual habit of installing PyPI packages to my user site directory (`~/.local`) no longer working due to the `externally-managed-environment` restriction. |
| 8 | + |
| 9 | +I consider it a perfectly valid and common use case to install some frequently-used packages (e.g. `regex`, `requests`, `mkdocs`) to my home directory so that they're conveniently available whenever and wherever I start a `python3` REPL shell. |
| 10 | + |
| 11 | +Fortunately, it's not hard to find a straightforward workaround once you know [this](https://docs.python.org/3/library/sys_path_init.html): |
| 12 | + |
| 13 | +> If `PYTHONHOME` is not set, and **a `pyvenv.cfg` file is found** [...] **in its parent directory**, |
| 14 | +> `sys.prefix` and `sys.exec_prefix` get set to the directory containing `pyvenv.cfg`, [...]. |
| 15 | +> This is used by Virtual Environments. |
| 16 | +
|
| 17 | +So if we can trick Python into thinking `~/.local` is a virtual environment, the issue resolves itself. |
| 18 | +It's not hard to figure out the required content for `~/.local/pyvenv.cfg`, much less writing it. |
| 19 | + |
| 20 | +```ini |
| 21 | +include-system-site-packages = true |
| 22 | +``` |
| 23 | + |
| 24 | +Of course an empty `pyvenv.cfg` will do the trick, but including system packages allows you to skip some redundancy between your user site and system site directories. |
| 25 | + |
| 26 | +We also need to ensure that Python is invoked from `~/.local/bin`: |
| 27 | + |
| 28 | +```shell |
| 29 | +ln -sfn /usr/bin/python3 ~/.local/bin/python3 |
| 30 | +``` |
| 31 | + |
| 32 | +However, `pip3` still complains about `externally-managed-environment`. |
| 33 | +This is because the Python interpreter that runs `pip3` is still launched from the system path (`/usr/bin/python3`) due to the shebang line. |
| 34 | +To fix this, just call `pip` with our desired path to Python: |
| 35 | + |
| 36 | +```shell |
| 37 | +# Assuming ~/.local/bin comes early in $PATH |
| 38 | +python3 -m pip install -U pip |
| 39 | +``` |
| 40 | + |
| 41 | +We can now inspect the new `pip3` command at the new location: |
| 42 | + |
| 43 | +```shell |
| 44 | +$ head -1 $(which pip3) |
| 45 | +#!/home/ubuntu/.local/bin/python3 |
| 46 | +``` |
| 47 | + |
| 48 | +Now this `pip3` command will happily install any requested package into `~/.local/lib` without complaining about environments: |
| 49 | + |
| 50 | +```shell |
| 51 | +$ pip3 install --user --upgrade humanize |
| 52 | +Looking in indexes: https://mirrors.ustc.edu.cn/pypi/simple |
| 53 | +Requirement already satisfied: humanize in /usr/lib/python3/dist-packages (4.12.1) |
| 54 | +Collecting humanize |
| 55 | + Using cached https://mirrors.ustc.edu.cn/pypi/packages/c5/7b/bca5613a0c3b542420cf92bd5e5fb8ebd5435ce1011a091f66bb7693285e/humanize-4.15.0-py3-none-any.whl (132 kB) |
| 56 | +Installing collected packages: humanize |
| 57 | +Successfully installed humanize-4.15.0 |
| 58 | +``` |
| 59 | + |
| 60 | +We don't need the `--user` flag anymore but I kept it as a habit of being explicit. |
| 61 | + |
| 62 | +## Using `uv` {#uv} |
| 63 | + |
| 64 | +Python `pip` has no idea when a package is installed and *why*, and in particular it does not know the difference between your package demands and what has been installed. |
| 65 | +If your `requirements.txt` undergoes changes and you want to ensure you have exactly the necessary set of packages installed, your best bet is to delete your `venv` and create it anew. |
| 66 | + |
| 67 | +So here comes [uv], a next-generation package *manager* for Python, whereas `pip` is at best a package *downloader*. |
| 68 | +As uv is designed for managing virtual environments within projects, some minor hacks are needed to make it treat `~/.local` as our venv, instead of creating another `.venv` directory. |
| 69 | +This is laid out in the [uv documentation](https://docs.astral.sh/uv/reference/environment/#uv_project_environment). |
| 70 | + |
| 71 | + [uv]: https://docs.astral.sh/uv/ |
| 72 | + |
| 73 | +```shell |
| 74 | +export UV_PROJECT_ENVIRONMENT=~/.local |
| 75 | +``` |
| 76 | + |
| 77 | +I also migrated my manual installations to a proper `pyproject.toml`: |
| 78 | + |
| 79 | +```toml |
| 80 | +[project] |
| 81 | +name = "home" |
| 82 | +version = "0.0.1" |
| 83 | +requires-python = ">=3.12, <4.0" |
| 84 | +dependencies = [ |
| 85 | + "jieba", |
| 86 | + "matplotlib", |
| 87 | + "mkdocs (>=1.6.1, <2)", |
| 88 | + "mkdocs-material~=9.7.6", |
| 89 | + "numpy", |
| 90 | + "python-telegram-bot==13.15", |
| 91 | + "regex", |
| 92 | + "requests", |
| 93 | + "tabulate", |
| 94 | + "yt-dlp", |
| 95 | +] |
| 96 | +``` |
| 97 | + |
| 98 | +Now we can simply run `uv sync` to have our desired packages ready with no fluff, and `uv sync --upgrade` to upgrade them when needed: |
| 99 | + |
| 100 | +```shell |
| 101 | +$ uv sync --upgrade |
| 102 | +Resolved 71 packages in 517ms |
| 103 | +Prepared 1 package in 447ms |
| 104 | +Uninstalled 2 packages in 57ms |
| 105 | +Installed 1 package in 45ms |
| 106 | + - humanize==4.15.0 # oops |
| 107 | + - yt-dlp==2026.3.3 |
| 108 | + + yt-dlp==2026.3.13 |
| 109 | +``` |
| 110 | + |
| 111 | +Because the environment variable `UV_PROJECT_ENVIRONMENT` is mandatory, I also created a Makefile to streamline command-line workflow: |
| 112 | + |
| 113 | +```makefile |
| 114 | +export UV_PROJECT_ENVIRONMENT = $(HOME)/.local |
| 115 | + |
| 116 | +.PHONY: sync upgrade |
| 117 | +sync: |
| 118 | + uv sync |
| 119 | + |
| 120 | +upgrade: |
| 121 | + uv sync --upgrade |
| 122 | +``` |
| 123 | + |
| 124 | +Now whenever I need another package inside my "home environment", I simply edit `pyproject.toml` here and run `make`. |
| 125 | +`uv` will ensure a consistent and sane package state, and I'll never worry about `pip` again. |
0 commit comments