Figure 1: The laptop that was having a bad day with NixOS 23.11

Figure 1: The laptop that was having a bad day with NixOS 23.11

More upgrade gotchas. Shucks. If everything goes well, this will be my last NixOS post. Read on to understand my frustration just a little bit more.

My main laptop is a Lenovo Ideapad Flex 5 — simple and cheap device. The keyboard stopped working in the early boot after upgrading to 23.11. The impact: I need to a USB keyboard around to unlock the device from a cold boot. Unfortunate for me, I forgot to bring one when I took the train to Chicago the other day. Instead, I had to spend a bunch of time troubleshooting.

§NixOS not discovering keyboard driver?

Figure 2: User typing on keyboard to unlock disk, but no effect

Figure 2: User typing on keyboard to unlock disk, but no effect

Whenever I was typing at the console to unlock the storage’s full disk encryption, the cursor didn’t seem to blink. This suggested perhaps the keyboard wasn’t being recognized.

I discovered the cause after booting a GRML live USB that was in my bag for a different project. I spent time looking at its lsmod output, and checking various details in /dev/input/input*/device/ including the description file. I found one description file that had the contents of i8042 KBD port. This description seemingly correlated to a i8042 module loaded in the Kernel.

Then I looked at the lsmod output again, grepping for lines containing i8042. I noted there is a ideapad_laptop module which depends on i8042. Bingo, this seemed like the module to load at boot. Might be fine to load i8042 directly, however ideapad_laptop seems like the driver for this laptop’s specific hardware, so I figured why not load it all.

Got lucky, turned out an older NixOS generation seemingly worked, so I didn’t have to figure out GRML networking, somehow get ZFS set up on it, and unlock the disks in GRML. Instead I just booted the old NixOS generation, fired up emacs, ignored all the errors caused by Emacs trying to load compiled elisp objects between major releases or something of that nature. nix-mode didn’t work either but that’s fine, fundamental-mode works too I guess. I added "ideapad_laptop" to boot.kernelModules in nix flake (commit), then ran a nixos-rebuild boot --flake .#.

Rebooted and viola! I could boot into 23.11 and use my keyboard!

§Lessons learned

To get into the broken NixOS install faster, I could carry on a live USB on my person. This way I can access my NixOS installation without further shenanigans. It can minimize friction caused through booting an older generation that messes with user data in frictionful ways (such as Emacs failing to load nix-mode in the older generation). GRML doesn’t ship with ZFS so it doesn’t quite meet this need.

nixos-generate-config is not infallible. It doesn’t include this i8042 driver for my laptop’s configuration and the initramfs doesn’t seem to load the driver without the above fix.

It appears I need to run NixOS unstable and test it on my machines in order to validate my installs prior to release. This is a lot of work and I’m not up for it. It doesn’t bring me value or joy. As of Jan 15, 2024, there are over 7k open issues, 5k open pull requests. That’s a lot of work blocked or waiting to be done. How do I know that my particular breakage will be addressed in any reasonable timeline? How do I know that my pull requests will be commented on in a timely manner? No promises, and therefore I am not demoralized from implementing further release testing on my end anyways.

§Next steps for my computing needs

Figure 3: My unpaid part-time job trying to write blog posts

Figure 3: My unpaid part-time job trying to write blog posts

It’s becoming more and more apparent I don’t want to play computer janitor every time I go to write an email or load a webpage. To this end I don’t think I’ll be using NixOS in the future. Need something with a committed testing culture and mechanisms in place to hear community feedback and take appropriate actions to correct direction / maintain focus on a singular vision. I could write a book about all my grievances… it’s besides the point… it’s bad energy and it’s hubris to think I could even change anything about the community, time to move on.

Debian Testing combined with Ansible and Nix might be sufficient. Or Gentoo combined with Ansible and Nix could also work. Gentoo now offers pre-built binpkgs so the installation experience might be about as fast as deploying a NixOS host from scratch. Bonus, while Nix offers some built-input parameters, there isn’t standardization on what those are like Gentoo USE flags, so Gentoo wins out in package customization in a consistent way.

Upgrading is a time commitment, NixOS can’t fix this. I dread the thought of upgrading to the next release, so my migration timeline will be by the time 23.11 is EOL (in about 5 months — 30 Jun 2024… yet another frustration with NixOS).

§What about you?

Does NixOS work for you? If you were to calculate the time spent per week learning how to use Nix/NixOS, fixing trivial issues that one takes for granted as fixed on most other distros, trying not to burn yourself out in nixpkgs, and asking for help but not getting any assistance… was it worth it?