Why do you use so many package managers?

7 July 2023

Package managers are a big deal. Every Linux distribution has one, and almost any other part of the OS can be swapped out without changing its character. Arch can use X or Wayland. Ubuntu can use GNOME or KDE. But Ubuntu can’t use pacman, nor can Arch use apt. The soul of a distro is its package manager.

A man cannot serve two masters; nor can a computer serve two package managers. And yet people, especially technically savvy people, do so all the time. Why?

Language-based package managers

Every programming language but C seems to have its own package manager. The practice is so pervasive that it almost feels natural. If you want to install python packages, you use pip. For JS packages, you use npm. For Ruby gems you use bundle. Why would you want it to be any different? I’m glad you asked!

You get all the same issues that anyone would get from mixing package managers.

This is not an abstract problem. One of the first things that happens to people who get interested in Linux is that they find a cool bleeding-edge package that’s not available in their distro. So they go to the Github page, see “install with pip3”, and open up Pandora’s box. This is particularly dangerous with new users who will resort to sudo at the first sign of trouble.

Library managers are designed for commercial programmers, so their needs are prioritised above all other concerns. And for a modern application developers, they usually work fine. Indeed, for packages that nobody but a software developer will ever use, there isn’t much of a problem. But generally useful software always has dependencies, and if those dependencies are only packaged for pip then you can bet the dependent will be packaged for pip before anything else.

But it’s cross-platform!

Making a piece of software “cross-platform” means to make it run on as many machines as possible, with no change to the way it’s built and packaged. Clearly, if all other things are held constant, then being cross-platform is better than not. It is reasonable for the maintainers of a program to make design decisions that make the software more agnostic about the host system.

But in this case, not everything is constant: the user is forced to accept additional complexity in order to run the program. In fact, the problem is asymmetric in two ways. First, the user is much less likely to be technically competent than the developer, and thus imposing a technical burden on the user causes more harm than imposing it on the developer. Second, a package might be developed by only a handful of people, but used by millions of them. So that wasted technical burden is spent among many more people than it would be if the developer were the one being burdened.

Oh, I almost forgot. Some operating systems don’t come with package managers. If that describes your OS, simply submit a bug report to them. I’m sure they’ll fix it.

But I need to control my environment!

When you’re writing software, the major thing you want is for your computer to get out of the way. In particular, nobody in the world wants to find out that they can’t build their Rails 7 hobby project because they happen to have a commercial Rails 5 project installed already. Hence, almost every language-based package manager comes with a scheme for building virtual environments. In fact, they usually come with several. There’s a couple for controlling the version of the language you’re using, and a couple of others controlling the packages you install in the current project. They all come with different syntax, and don’t you dare mix them!

But this only solves half the problem. What happens when you need to switch out your database? Or your web browser? What happens when you want to write software that depends on a different version of X? Or if you want to work on the kernel? Oh, sorry, I forgot - everyone’s a web developer. So to switch out your database, you use docker, right? Or a VM. Or something equally powerful, which is perfectly capable of installing your libraries too! In fact, it’s extremely common for people to use something like pip inside a docker container. Why not just use the included package manager? Oh, because nobody wrote packages for it. Cool!

Said differently, virtualisation is a legitimate problem that should be solved in full generality. Having your environment tied to a specific programming language just pushes you down a path of duplicated tools and extra complexity.

But my dependencies don’t exist in apt/pacman/yum!

This is the only argument that I’m actually sympathetic to, but it isn’t an argument in favour of language-based package managers - it’s just describing a problem they cause.

That’s not entirely fair. If you only ever write python, and you don’t want anyone outside the “python ecosystem” using your software, then it’s much easier for you to only write one package rather than one for every linux distribution under the sun. Since a universal standard for package management will probably never exist, the only realistic solution is machine translation. In other words, it must be possible to automatically parse a package definition, so that it can be automatically and reliably translated into other formats for each release. This sort of thing does already exist with e.g. bundix in the Nix packaging system.

If we have that, then feel free to write packages for whatever system you like. But don’t use more than one package manager on your running system. And since language-based package managers cannot support a full operating system, that rules them out as viable options.

It’s not that complex, I understand it just fine!

Yeah, for now you do. The issue is that a lot of the knowledge you gain about one ecosystem doesn’t translate to the others. If you’ve been working in Ruby your whole career, and you introduce node to a project, or you write a little python utility, it’s not enough to know the language. You have to know all about how to install packages, what syntax do you use for version locking, how are things cached, how do I debug them?

And that happens every time you switch. Maybe you only work in one environment, ever. But in my experience, if you want to learn a new language or framework it’s a huge headache before you can really understand what you’re doing. It’s trivial to get up a “hello, world” app - but you have that and no understanding of how your computer is working. I’d much rather have the understanding.

After all, it’s not like Python libraries are magically different to other sorts of software. It’s just code. Install it somewhere!

Application plugin managers

Ok, I admit it. I use vim. I use it at work, and I use it at home. I use it as root. I even use it on my VPS. And I want it to be the same program on all of them! That means I want the same colourscheme, the same keybindings, and the same plugins. I can’t even remember how I set that up the first time. What plugins do I even have installed, actually? There should be a single place I can check.

“That’s easy.”, I hear you snort through the screen. “It’s just in your dotfiles. Don’t you have a git repo with your dotfiles? Oh, mine’s on my Github. Or is it on BitBucket? Then you just need a short script you run when you’re on a new machine. Oh, my script doesn’t work? Well it works for me…”. Stop it. You’re being a package manager. Whenever you catch yourself managing packages, stop it, and get your package manager to do it instead. It’s better than you.

At least in this case you can track what’s installed, probably. If it’s in a text file somewhere, you can at least store that file, and then download it when you’re on the new machine. You might need to write a script, but maybe someone will write it for you. What about when it isn’t? What about browser extensions, or VS code plugins (perish the thought)? Then you’re really screwed, you’d better be willing to sign up for an account. And don’t get me started about storing the passwords…

What’s the alternative?

For a long time I had no idea. I was using Gentoo for a couple of years, and I got really excited about their concept of SLOTs - the Gentoo package manager’s scheme for allowing, say, multiple Python versions to be built on the same system and swapped in or out. But when I looked into using it more widely, it turned out that the process was quite cumbersome.

Nowadays I use Nix, and it feels like they are going in the direction I want. In particular, there are entire translation layers for things like Ruby’s bundle, which takes a Gemfile (list of packages) as input, and converts them to the Nix version. Out of the box, you can have arbitrarily incompatible versions of software installed on your system - as long as only one of them is active in your current environment, obviously. I plan to document some of those examples in the future.

Before you email me that XKCD standards comic: no, I’m not saying “Just trust me guys, as long as everyone migrates to Nix, we’ll all be livin’ off the fat o’ the land!”. I don’t care whether you migrate to Nix or not. But Nix derivations are machine-readable, so it should be possible for you to use them, too. I should sanity check that.

But whatever you pick. Just pick one.