Getting into NixOS

A few weeks ago, I was chatting with one of my friends, and the topic of photo hosting came up. As the resident photographers of our friend group, we have accumulated a sizeable collection of photos of birthdays and other group events, but they have largely languished in Lightroom or Darktable libraries.

After comparing cloud offerings, we came to the conclusion that building a home server could actually be competitive, especially considering that it could also be used to run various other services. A spreadsheet and a few eBay purchases later, I now have a server humming away in a spare closet.

As a result, the cloud server I have been using for the past six years is in an awkward position. Many of the non-essential services can now be moved to the home server, and the few things that require the uptime and stability guarantees of the cloud could run on a far cheaper instance somewhere else. Admittedly, I probably could've moved things over even earlier, but there has always been one issue: server migrations are a giant slog.

A personal server is in many ways similar to a living space, except you never have anyone over and you never seem to run out of dishes. When you eventually have to move, you suddenly realize that the place is a mess, all your cables are tangled up, and you have no idea where you put those tax documents.

The beautiful cottage at hand here runs a fairly standard Ubuntu Server install, but it has grown a few warts over the years. Most everything dates back to a time before containerization was widespread, and a lot of the custom services are Python applications that were written seemingly before anyone had Python dependency management worked out. Additionally, a Postgres database contains data for a number of services, including a somewhat bodged together mail server.

Badly specified shared state is pretty much the name of the game here. There is a stray LXC container -- because I was too quirky to use Docker, I guess -- but that is the only potentially clean extraction here.

Biting the bullet

At this point, it really isn't worth paying fifteen bucks a month for, though. And if I have to migrate this mess anyway, I might as well do it properly, right? We live in the future, we have the technology, etc.

To be fair, we had the technology back in 2018. Back then, the canonical solution was probably Docker and Docker Compose, and they are still perfectly reasonable solutions, depending on your requirements. For some reason, the whole "cloud native" computing stack always rubbed me the wrong way, though. There is something very corporate and techbro about the whole thing, obviously, but it turns out it does solve real problems! Who knew! (Everybody but me.)

And so, back in 2018, I definitely did not get on the containerization hype train. Instead, I thought I probably just needed to write some documentation. Y'know, just write down what services should be running, what their configuration should be, how things are set up. Then, when the time comes I have to migrate and set this shit up again, I'll know what to do.

Of course, that documentation is now woefully out of date. It turns out you will forget to update it, and when you do, you will miss things. Without organizational processes to keep documentation and source in sync, they will inevitably drift apart.

Over the years, I have become more comfortable with various solutions to server provisioning. I've used Terraform and Ansible on the job, doing server administration for a university web development course, and I think they're reasonable options, especially if you want the result to be a bog-standard Linux server without any strange quirks.

That said, Ansible doesn't really fix my problem, unless I religiously make sure to edit my Ansible plays and associated configuration instead of making changes to the server directly. Which, given the above track record... doesn't seem likely.

Ideally, there would be some automated way to keep the system state consistent with its description. Perhaps some kind of meta-configuration, which forms a single source of truth that a complete system could be derived from.

Nah, surely that's unrealistic...

Nix and NixOS

Of course, given the title of this blog post, the astute reader may already surmise that there is a solution to these problems: Nix.

Nix has recently seen a surge in popularity, but the project is actually remarkably old, dating back all the way to 2003. After reading the original 2004 paper on it1, it is honestly surprising that it stayed relatively obscure for so long - here was an elegant solution to real-world problems, which took nearly two decades to become mainstream!

I vaguely recall reading something about it on Reddit sometime in the mid-2010s, but I didn't really get it until recently, after reading Amos's fantastic Building a Rust service with Nix series. Even then, I didn't seriously think about using it until a few weeks ago, when I began formulating this project.

Essentially, NixOS allows you to declaratively specify the configuration of a system. That is, you describe what the state of the system should be (which users it has, what services are running, how the network is configured), and NixOS figures out how to make it so. The NixOS configuration is modular, and tightly integrates with the underlying Nix package manager, whose official repository (nixpkgs) already contains over 80,000 packages, which means you generally don't have to start from scratch when integrating a piece of software with NixOS.

Sounds great, how do I use it?

Well, if you're interested in packaging software with Nix, I recommend reading the aforementioned series on building a Rust service with Nix. Even if you don't use Rust, the concepts underlying Nix are still relevant and very well explained. This is also one of the few introductions that immediately makes you use Nix Flakes; the flake system is a big improvement on traditional Nix, but unfortunately the Nix project is in a bit of a transitional period, and a lot of Nix resources still describe the old way of doing things.

As for NixOS: the official manual does a pretty good job of describing the installation process. Sadly, the default NixOS install has not yet been "flakified" either; the NixOS & Flakes book fills the gap and walks you through flakifying a new NixOS install. It also has some good pointers for learning the Nix language, which is used to configure NixOS as well as write Nix packages.

Although Nix and NixOS are very powerful, and ultimately an improvement on the traditional package management and system configuration approaches, they do take some getting used to. As with many open source projects, documentation can sometimes be scarce, and the road to understanding Nix can be frustrating (it has been for me, anyway). That said, in my opinion, the benefits of the Nix approach easily outweigh the initial investment in learning it.

Tips from a fellow beginner

As I have been migrating my server configuration to NixOS, there are some things I have learned that may be useful to fellow Nix beginners.

NixOS modules and Nix packages

In addition to "raw" packages, the nixpkgs repository often also contains corresponding NixOS modules that can be used to configure a service. For instance, the Nix search tool will tell you that you can add the bind DNS server package to your NixOS configuration by adding the following snippet:

environment.systemPackages = [
    pkgs.bind
];

This works, but probably isn't what you want - this will install bind but does not provide a way of configuring it. Instead, what you probably want to do is configure the bind NixOS module. If you are lucky, the NixOS manual will have a section on configuring the software you're interested in (e.g. PostgreSQL). If not, the community-run NixOS wiki might (e.g. Nginx). Otherwise, you are unfortunately stuck reading Nix source code and using the NixOS options search, as is the case for bind.

Ideally, each NixOS module would come with easily accessible configuration instructions, but we aren't there yet.

Packaging Python with Nix

The Python dependency management story has been a mess for many years. The solution I have settled on is using Poetry with poetry2nix, which generally works but has some rough edges. They seem to be more or less inevitable given the state of Python packaging, unfortunately. For a complete example, see this Nix flake for one of my custom Python services.

Secret management

Many people use a git repository to store their NixOS configurations, with many of them even being public. This presents a problem, of course, when you need to pass sensitive information (tokens, passwords, et cetera) to services in the NixOS config. There are various solutions to this, many using some form of encryption to safely hide the secrets, automatically decrypting them at runtime. I can vouch for agenix, although I haven't used any of the alternatives.

Update (04/2024): the big migration is done!

All in all, it took me about a month to migrate everything to the new server. Pretty much all the software I used, other than the bespoke things I wrote myself, is available in the form of NixOS modules, and the migration was really just a case of putting the persistent data for each piece of software in a standardized location (subdirectories of /data, in my case), writing a NixOS module configuration, and running nixos-rebuild. The only thing I gave up on was my mail server; although I'm sure it could be made to work, the headache of redoing a postfix and dovecot configuration was psychologically too much for me to handle. Instead, I switched to maddy, and I haven't looked back.

Occasionally, there were annoyances that were difficult to fix, and required deep dives through NixOS Discourse, various manuals and blog posts to figure out. That is a pity, but also just a one-time time investment. And I'm very happy with the result: a server configuration that is finally easy to edit and keep track of.

1. See Dolstra, E., De Jonge, M., & Visser, E. (2004). Nix: A Safe and Policy-Free System for Software Deployment. In LISA (Vol. 4, pp. 79-92). ↩