Package Management in Python
Python packaging is a strange beast; the ecosystem of packages is one of the things that makes Python both so amazing and frustrating.
Python was not originally designed with packaging in mind. While ad hoc sharing of code regularly occurred from its beginning, a built-in method to package Python code via distutils was not available until Python 1.6.1 in 2000 (a full 9 years after Python was first released). Metadata was standardized in PEP241 in 2001 and PEP314 in 2003, and the official third-party software repository PyPI finally came in 2003 as described in PEP301 (meanwhile, Perl's CPAN was released in 1995). Automatic declaration and installation of dependencies came with setuptools in 2004. Packages could finally be hosted on PyPI in 2005, but the commonly used pip package installer didn't arrive until 2008 (replacing easy_install).
Python is used very differently than many of the languages that came before it. Most programmers are not writing long-lived, well-funded applications, but are quickly gluing together existing code (I personally write a lot of code that interfaces with C++ and Fortran applications). It is particularly popular in the scientific community for solving specific problems and then disseminating these solutions; as such, there are often many conflicting dependencies and less focus on maintaining high-quality stand-alone packages (let alone a focus on consistently updating code for the latest versions of dependencies). This leads to particularly fragile ecosystems (as pip installs packages system-wide) and conflicts between package versions as programmers install large numbers of packages with complicated dependency webs.
Virtualenv was released in 2007 to provide local installation of packages. This allowed each separate script and application to easily load packages from PyPI without worrying about conflicting packages on the system. However, the ergonomics of it were less than great, and few Python programmers used it. The additional difficulty of distributing multi-language packages was addressed by PEP 425 and PEP 427, which were introduced in 2012 to allow targeting of specific platforms and distribution of wheels containing pre-built binaries (replacing the older egg format). Conda was released around the same time to provide a language-agnostic cross-platform environment manager, and many packages and applications switched to it (Jake VanderPlas has a good write-up on the need for Conda, and Python's BDFL Guido van Rossum encouraged Conda's development). However, the growth of Python packages, their interleaving dependencies, and the splintering of package managers and environments inevitably led to issues.
Conda and its associated package repository quickly became popular for providing a simple way to download mixed-language packages. The associated conda-forge is also popular for allowing distribution of community developed packages. However, the lack of lockfiles made it impossible to guarantee reproducible builds without specifying the precise version in the environment file (e.g. package=1.2.3-patch45). Conda was also slow at solving for a viable combination of packages, so Mamba was released in 2019. Mamba provided parallel package downloads, better dependency solving algorithms, and a C++ rewrite. However, since it tried to be a drop-in replacement for Conda, it did nothing to fix its other shortcomings.
Various other package and environment managers have come and gone, including Pipenv, Poetry, Hatch, PDM, and many others. However, most of them were slow, relied on the system Python, didn't provide lockfiles, or made other mistakes (e.g. Poetry's (in my opinion) ill-fated attempt to encourage the use of caret notation with semantic versioning, ^1.2.3). Many of these were also hampered by attempts to be marginal improvements on the status quo, instead of ground-up redesigns of how a developer friendly environment and package manager should be built.
Modern package managers
As Python has struggled with packaging, various languages released integrated support for advanced package management, such as golang1 and rust.2 The package managers in these languages (go modules and Cargo) provided simple ways to build, test, and run projects across environments, all while not requiring system-wide library installation. These principles have influenced many subsequent languages, and the development of the two leading modern Python package managers (ironically, both of which are not even at version 1.0).
Pixi (from prefix.dev) was released in August 2023. It provides an isolated environment for package installation and running. It has the ability to easily install packages from both PyPI (via rip and then uv) and Conda (via rattler). Multiple environments can be set up, allowing the separation of development tools from package dependencies. The platforms of interest (e.g. linux-64, osx-arm64 ) can be selected to ensure that the lockfile contains the correct combinations of packages to build on all requested platforms. Pixi also provides simple ways to run tasks via the pixi run syntax, simplifying the running of complicated commands (e.g. pixi shell -e dev ruff check . --fix to pixi run lint).
uv (from Astral) was released in February 2024, explicitly trying to be the "Cargo for Python". uv eschews installation of Conda packages and only installs packages from PyPI and the recently released pyx into isolated package-specific environments. uv also supports CLI, GUI, and plugin entry points (e.g. uv run lint can call ruff check . --fix in the environment).
Selecting a package manager
Both Pixi and uv are excellent package managers, but both aim to do slightly different things. Pixi is needed when packages require installation from conda, as is common in scientific Python development. For this reason, Pixi is the package manager for most of Rowan's applications. However, if a package is Python only and all dependencies can be found on PyPI, uv should probably be preferred due to its significantly faster solution times and excellent build system for distribution on PyPI. Both packages technically have not reached version 1.0, but having used both for a while now, there is nothing I miss in any previous package managers, and Pixi and uv will only continue to improve.
Of course, there is more than just selecting a package manager; one should also consider setting up their Python package with something like uv-cookiecutter or pixi-cookiecutter, but that is for another post.