gay robot noises

I use full-disk encryption on my laptop because I'm a bit of a control nerd like that. And I didn't write down the password anywhere because that meant that anyone who got that password would probably also be able to get my laptop. Finally, I had a shorter password for my user account since I type it frequently (sudo, unlock screensaver, etc). Which worked fine... until I had a month or so where I never turned my laptop all the way off. And then when it finally ran out of power and I turned it back on, I realized I had no clue what the password was any more. Shit. I tried everything I could think of in every variation, and it didn't work.

Fortunately, I have daily backups at 4 AM via borg running in a systemd timer, both to a server in my office and to rsync.net. And I hadn't used the laptop all day, so my data would still all be there.

Unfortunately, those backups are also encrypted using a passphrase that I store in a file on my laptop, and I never even bothered to memorize it. I thought I wrote it down, but it wasn't in any of my usual "important documents" locations.

Fortunately, that keyfile is stored in my nixos-config git repo, which I push fairly regularly to codeberg.

Unfortunately, it's encrypted using agenix to my SSH key (because storing unencrypted secrets even in a private repo is a terrible idea), and the SSH key wasn't backed up.

Fortunately, it was also encrypted to my SSH key on my old laptop, whose password I did mercifully remember. And I was able to plug it in, turn it on for the first time in months, and borg mount the local copy of my backups. And since my NixOS config is in source control and I'd recently pushed it, I knew I'd be able to recover all my system configuration. So I borrowed a USB drive, wrote the NixOS minimal installer ISO to it, and flattened the drive.

Lessons learned

I don't get to do post-mortems much, so I figured I'd do one for this. :)

What went wrong

• Not using the FDE password often meant that I could easily forget it, especially since the human memory has this very annoying and unpredictable retention policy. Solution: set my user account password (which I enter at least once a day when I wake my laptop from sleep) to the disk password.

• Neither the FDE password nor the Borg passphrase were written anywhere. Solution: I stored both my backup passwords and my user password in my password manager (a self-hosted vaultwarden instance). Forgetting the password again is a much more relevant threat than someone stealing my phone while it's unlocked and also stealing my laptop.

What went right

• I have a status indicator in waybar that turns red if I haven't had any successful backups recently (it's red right now since it checks systemd journals, and this new install doesn't yet have its own set of SSH keys). I make sure not to let it get red. So I have high confidence my backups are succeeding. I don't test the restore process as often as I should (though I have tested it before), but it worked fine.

• Versioning my NixOS config means that, after a bit of bootstrapping (my config uses flakes, and I wasn't sure if the installer's nixos-install command was flake-aware), I was able to get to a setup that was basically identical to what I had before.

• I back up my entire home directory; the only exceptions are things like node_modules or Rust target directories. This means I back up all my .config files, so all the things that aren't covered by home-manager I can just copy back from the backup.

• In general, the entire process took about 4 hours, and most of that was either me fumbling with the initial NixOS install or waiting for the backup to copy; there are a couple multi-gigabyte files I didn't expect, including a cache for the parcel build tool that took up something like 40 gigabytes.

Some amusing mistakes I made

• I use Colemak-DH as a keyboard layout. The installer USB has loadkeys, which has a colemak layout, but several keys (d, h, m, g, and a few others) are in the wrong position. If I was thinking properly, I would have just plugged in my keyboard (which does the keyboard layout in hardware), but I was a bit stressed.

• All the instructions for installing NixOS I was using assume your target disk is /dev/sda. When I went to gparted the disk, it complained that /dev/sda was in use. I figured that it had just automounted it and ignored that. It turns out that my laptop uses NVMe, so the disk was on /dev/nvme0. I was formatting the installer USB. Which wouldn't be a problem, except...

• I set up LUKS disk encryption again, and nixos-install doesn't automatically generate configuration for that. So the generated hardware-config.nix file didn't have any settings saying "hey, there's an LVM volume you need to decrypt", which meant that the machine hung on boot. Which meant I needed to then reformat the installer USB since I'd wrecked it.

• I ran the installer using the minimal system the installer generates (as well as NetworkManager so I could get on wifi) because I wasn't sure if the installer would be able to handle all the advanced things I do in my configuration. After the initial boot, I cloned down my NixOS config and ran sudo nixos-rebuild switch. And as it was finishing, the connection dropped. Because I was running this from my old laptop over wifi (to avoid dealing with the keyboard layout issue), and that restarted NetworkManager.

• systemd also went into recovery mode for some reason. And when I restarted, it failed to come up, because I'd copied over the hardware-configuration.nix from the old install, which had completely separate partition UUIDs. Fortunately, this time I was able to just boot it into the first setup, so I didn't have to boot off the rescue USB yet again.

Other trivia

• Since I don't yet have the keyboard layout set up when I'm entering my disk password, I actually have LUKS set up with two keys: one that's the password proper, and one that's the password typed with Colemak-DH finger motions but on a QWERTY layout. So I can decrypt my disk both from the built-in keyboard and from a 'hardware Colemak-DH' keyboard. I'm sure there's a way to set up the console so that it loads Colemak-DH early on (or I could just reflash the laptop firmware so that the laptop keyboard is 'hardware Colemak-DH'), but this approach is funnier to me.

• The initial launch of emacs took about as long as the entire nixos-rebuild switch that installed all of my binaries. This is partly because straight.el doesn't parallelize its git clones and partly because it was native-compiling a bunch of those packages (as opposed to nixos-rebuild, which pulls prebuilt binaries). This isn't a huge deal, but it is silly.