NixOS Migration

Recently I have begun migrating my workstation and laptop from Gentoo to NixOS. There are a great deal of tradeoffs between the two operating systems. Before going into the details, consider where I’m coming from and why I moved away from Gentoo below.

§Why was I running Gentoo on workstations??

This is my heuristic for a good operating system:

The Distro must provide facility to modify system packages and maintain their modifications in sync with the upstream distro.

This eliminates a lot of “popular” distros as it is not a an ergonomic process to work with RPM spec files, the Debian build system, and even Arch is not very practical in this regard¹. The remaining candidate Distros are the likes of Gentoo, GuixSD, NixOS, Alpine, and using linuxbrew/pkgsrc/other “ports-like” systems.

Anyways, back in 2018 I migrated from Arch to Gentoo. It went super well! I gained a lot of functionality. Most importantly I could patch any package I wish without risking dealing with library packages breaking my packages on major revision bumps. If you drop a /etc/portage, /var/lib/portage/world, /var/lib/portage/world_sets in a fresh Gentoo install, then rebuild world, you’re set. In other words, the package configuration is declarative in Gentoo. You specify what you want in a bunch of text files, and you know exactly what you’re getting and why. In fact, you can review my Gentoo configuration files in the GitLab.com winny-gentoo-ops namespace.

§Some Gentoo Gotchas

There are a few downsides with Gentoo I like to mention. There are no binary packages publicly available. This is a half-issue, as I built binary packages for all my installed software. Then I was able to install the same exact binary packages on my laptop. Still, it can take a bit of time to build packages.

The community in Gentoo is very kind and welcoming. There is presence on Discord, IRC, Forums, Mailing Lists. If you have a question you can usually get an answer. On the other hand, when trying to get bugs fixed, there aren’t enough people with commit status responsible for the ebuild repository. This results in a many bugs left unaddressed for months despite well-tested patches on the Bugzilla tracker. Due to lack of movement of bugs and PRs, myself and perhaps a portion of users don’t really bother contributing patches back anymore. It’s not a good use of time. It’s far more time efficient to drop patches on my own systems and call it a day. One should always report bugs, but I don’t hold my breath about them being fixed in the official repository.

The above also applies to contributing back to third party overlays - a lot of maintainers don’t really care for contributions and won’t overtly say that, but you see it in how they triage their PRs. They just want you to go away and do your own thing. Some aren’t like that, but the majority feel this way. I guess it makes sense, given the reason they use Gentoo is the similar to my own needs - ease of customization and ease of package maintenance.

§Why NixOS??

Now that you understand why I have reservations with using Gentoo for my workstation and laptop, I’ll contrast with the NixOS project.

First, the contribution model for NixOS is clearly defined and while there are a lot more open PRs and issues, there are also a lot more people triaging. This means you usually have tight feedback loop on pending contributions. There is a higher chance of upstreaming your fixes to the Distro on NixOS than Gentoo at this time.

Second, the package selection on NixOS is insanely diverse. According to Repology, there are 9195 unique packages in nixpkgs stable 22.05, followed by Raspbian Stable, Ubuntu 22.04, Debian 12, and FreeBSD.² This means I’ll invest less time packaging software, instead spend more time bugfixing and using my system.

Third, ZFS support is baked into the system. This is sort of a minor one, but any distro that makes you work to get ZFS support (e.g. having to take extra steps to enable/maintain it) is not good for ZFS adoption. I actively discourage using ZFS on such distros because the risk of messing everything up is just too great. Use something else like LVM2+Ext4/BTRFS and have less pain. On NixOS you simply just specify you wish to use ZFS and that’s it.³

Fourth, NixOS feels a lot more maintainable in general. As one specifies the majority of the system configuration in /etc/nixos, most of the important configuration is easy to version and track. Troubleshooting errors from nix feels a lot easier than browsing the portage codebase… the portage codebase feels less than ideal and hard to follow - it has a lot of history.⁴

Building upon the above, the lifecycle of NixOS versus Gentoo is a lot less work on the user side. On NixOS you usually just run nixos-rebuild switch --upgrade to update. On Gentoo you need to do a bit of gymnastics to ensure your system is upgraded to a sane state. Please see this script I wrote to automate the process. If I didn’t have the script I’d forget steps.

§Assessing NixOS viability

Still reading? Great! You’re a pretty cool reader, by the way. :)

To prepare for the migration, I tested NixOS installations within libvirtd. First was a simple Ext4 rootfs, followed by a BTRFS rootfs, then a ZFS rootfs. Then I went a bit more off the wall, tried Ext4 in LVM2 in LUKS - works great. Now what impressed me is I was able to also enable ZFS encryption on the root filesystem. This is the ticket - I can use ZFS and discard the encryption keys when I am ready to reuse the hardware.

Quick aside: as I played with the configuration, one very amusing tunable was discovered: services.xserver.desktopManager.cde.enable. Enable this, then reboot your system, enjoy! Yes this is a screenshot from my VM!

During the above testing, I found it a bit frustrating using the NixOS Gnome or KDE live media. I also don’t use qwerty so that’s a bit of a extra work to set up dvorak every time I boot the live media. After a quick Google, it turns out it is possible to create one’s own live media without much effort. In fact from start of reading the Wiki page to creating my custom live media elapsed about 1 hour - 30 minutes of which was the computer spinning its wheels compressing the squashfs. See my barebones dvorak, XFCE, Emacs livecd configuration here. Since creating this custom live media, I haven’t used the official media - it’s just so comfy.

I invested a bit of time when running NixOS in a testbench to port over most of my package selections from Gentoo. During testing I found a few packages that were missing… they might be my first packages to contribute back. I also found one bug that was promptly fixed by a caring maintainer.

§The migration

See also the Installation guide in the NixOS Manual.

The migration was fairly straightforward. I made a full disk image of my OS SSD using GNU ddrescue and saved it to an external HDD. This served as a fresh backup, though I had other local and offsite backups to refer to in case the external HDD fails. After verifying I can access data off the external HDD, I went ahead and wiped the SSD using blkdiscard.

Next I created the GPT partition table, a EFI and /boot partition, then the rest of the disk was set up for ZFS. Here is the partition listing:

[root@stargate:~]# parted /dev/nvme0n1 print
Model: Samsung SSD 980 PRO 1TB (nvme)
Disk /dev/nvme0n1: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size   File system  Name   Flags
 1      1049kB  1000MB  999MB  fat32        boot   boot, esp
 2      1000MB  1000GB  999GB               nixos

Next I created the ZFS pool. According to zpool history rpool, these are the commands executed to create the ZFS Zpool and ZFS Datasets:

zpool create \
      -O mountpoint=/ \
      -O encryption=on \
      -O keyformat=passphrase \
      -O keylocation=prompt \
      -o ashift=12 \
      -o autotrim=on \
      -O acltype=posixacl \
      -O xattr=sa \
      -O relatime=on \
      -O canmount=off \
      -O compression=zstd \
      -O dnodesize=auto \
      -O normalization=formD \
      -R /mnt rpool /dev/nvme0n1p2
zfs create -o mountpoint=none rpool/nixos
zfs create -o mountpoint=legacy rpool/nixos/nix
zfs create -o mountpoint=none rpool/userdata
zfs create -o mountpoint=legacy rpool/userdata/home
zfs create -o mountpoint=legacy rpool/userdata/root

All Datasets are encrypted by a passphrase. Note I chose to enable ZSTD compression out of the box for all Datasets. Compression reduces the amount data that’s written to the storage device, with a slight increase in CPU load. This tradeoff is usually worth it because disk IO is usually orders of magnitude slower than the compression/decompression CPU cycles.

Once the Datasets are created, be sure to mount them using an incantation like mount -t zfs POOLNAME/DATASET MOUNTPOINT. See the ZFS wiki page for more detail.

After all filesystems are mounted in the installation prefix, I ran nixos-generate-config --root /mnt to generate the configuration file skeleton. I cloned the git repository where I stored my testbench VM configurations, and dropped in my common winston.nix file(s) for use on all my machines. These configurations installed all the packages I want, set up XFCE autologin, and so on. After adjusting the config files with my favorite text editor, I ran nixos-install, entered a new root password. Before rebooting, I needed to set my user password, so I ran nixos-enter to automate the chroot steps and run commands within the new NixOS environment without rebooting. Finally I set the password for my user within the chroot: passwd winston.

After completion I rebooted, typed in my ZFS encryption passphrase, and was met with an auto-login XFCE4 desktop. I think that was the fastest OS installation I’ve done in a long time. It took about 30 minutes, most was downloading tens of GiB of packages (I run a bloat-y OS I know !).

Here’s an obligatory neofetch.

§First NixOS upgrade ever

Shortly after I installed NixOS, 22.05 was released. I held off for a few days until I felt the system was configured more-or-less as I wished it to be for the hardware. It turned out all one had to do is change the Nix channel (aka package repository) then run a command to rebuild the system. See Upgrading NixOS in the manual.

There were a few packages that got moved to different names. Nix told me about them, though it did not attempt to auto-migrate these obsolete package names, so I still had to edit the .nix files. Ah well, can’t win them all :).

One surprise was change in behavior in the nix package manager. nix search no longer worked, instead printing out a warning you have to enable experimental features. I had to add the following to my configuration.nix

nix = {
  extraOptions = ''
      experimental-features = nix-command flakes
    '';
};

The behavior of nix search had changed a bit, so one now needs to use nix search nixpkgs thing instead of nix search thing. Much to my surprise, the release notes did not mention this. See the version from June 2nd; compare to current. Thankfully a thoughtful developer reworked the release notes recently, so other users will know which nix release notes to also review (yay!).

There was one other minor issue caused by a setting being renamed or the like. That took a bit of research to determine how to port that knob to the new version of NixOS. I don’t recall which knob it was. It was confusing but I figured it out.

A quick reboot and everything is great as it was previously!

§ZFS Snapshot gotcha

I had a funny experience with enabling ZFS Automatic Snapshots. As it turns out you need to do a bit more than add services.zfs.autoSnapshot.enable. Indeed, you need to set a property on every ZFS Dataset you wish to be considered for automatic snapshots. In my case I just had to run zfs set com.sun:auto-snapshot=true rpool to enable automatic snapshots of every Dataset.

This is sort of my fault, as I did not do enough due diligence. It’s mentioned in the above option’s documentation, though I assumed nix would complain about a nonsensical configuration. In any case, I filed a GitHub issue to track this gotcha.

§Secondary LUKS volume

This was another funny NixOS gotcha. I kept asking in Matrix about how to enable LUKS for a second volume and nobody was familiar with how to answer that question. There was discussion to enable LUKS for the root filesystem. After reviewing the source code for luksroot.nix, it was evident this was not what I wanted. It took a bit of researching on Google and the Nix Discourse to come up with a solution and it’s kind of surprising.

The solution: just use /etc/crypttab like most systemd distros. Given most of NixOS is about wrapping up settings into a unified configuration language, it feels rather goofy to just drop in a standard configuration file. The way to do this is add something like this to your NixOS configuration:

environment.etc.crypttab = {
  enable = true;
  text = ''
      dmMapperNameHere UUID=b8104880-1fbf-41b5-9a90-2a4f1cc5d6de /secrets/your.key luks
    '';
};

Then nixos-rebuild switch. Your LUKS volume should now be automatically unlocked by keyfile.

§What’s next?

Currently, this workstation has automatic local snapshots, but needs off-site backups implemented. Previously I have used borgmatic to manage borgbackup repositories. I have one repository on a local NAS server and another hosted at borgbase.com. I think it would be more ideal to use zfs send because it’s much faster, especially for restoring from backup.

My laptop, still runs the old Gentoo configuration I shared between this workstation and the laptop. It seldom compiled packages locally because I bulk-built packages on my workstation. Now I’ll have to build packages on my laptop until I migrate. Maybe I’ll do the migration this week… no time like the present!

I need to learn a lot more about Nix and NixOS. I don’t know how profiles, flakes, or much of the internals work. I just know how to read source code and have some prior knowledge of how this system generally works (after all it’s just a Linux system with a lot of special sauce on top).

The ultimate test will be to do like I do on Gentoo; locally modify a system package without much fuss. Nobody is the wiser. Stay tuned for content about packaging software on Nix.

§Final Remarks

(XKCD, CC BY-NC 2.5)

Dear reader, if you wish to give NixOS a try, I strongly recommend ignoring much of the advice you get from experienced NixOS users. You may find them giving you good recommendations, but they are usually advanced stuff, like using home-manager, flakes, profiles, etc. All you really need to know is how to add stuff to /etc/nixos/configuration.nix and how to read error traces. Once you get a working NixOS system and you’re productive on it, I recommend exploring these more advanced topics, like I am doing in the near future.

I’m looking forward to streamlining more of my day-to-day computing. Gentoo was fun, but I need to re-invest some of the time and energy into other endeavors. One place where I am looking to continue to use Gentoo is specialty hardware and Retro-computing. Some distros have been bumping their minimum x86 supported CPU recently. This is starting to impact my Thinkpad x31 hobby laptop.

I have a challenge for the reader, try installing NixOS in a VM. Maybe deploy a website with it. See what you think for yourself. Maybe it’s not for you, but it’s important to be aware of these growths in computing.

§Thanks

Thank you to bard and haavard for copy editing this post. It was chock-full of typos!