Running Nix Infect on My Dedicated Server

Recently I rented a new dedicated server in a Miami Data Center to have a bunch of remote compute at my disposal to use from things like my phone and Macbook Air¹. I’m not sure yet what I’m going to do with it, but I think at least running this website or running a Mastodon instance would be options.

And because I love NixOS that is what I was planning to run on it directly (“bare metal”). I thought of using Proxmox, but figured that NixOS is probably better for what I want anyways and I could always install Proxmox later. That said, NixOS isn’t really a supported OS by my provider. They do support Debian though.

Thankfully, the nix-infect project exists for situations like this where you want to use NixOS, but the provider doesn’t easily let you upload your own machine images… or it’s a physical machine.

Maybe this could be solved easily with a support ticket, but I felt I didn’t really need the help as also didn’t want the hassle².

I did run into some pretty serious issues, but eventually I got it to work. First I had to “brick” the machine a few times because of an important configuration option in nix-infect that is disabled by default, except for Digital Ocean VMs (and me!).

How does Nix Infect work? Link to heading

Nix Infect is really simple: it’s just a bash script. Although it being bash script did make it difficult to correctly read the first time, it is nice because it’ll work pretty much anywhere and debugging problems with it is easy.

The first things it does is make sure it is running as root and that the required tools (curl, ip, awk, etc.) are installed. If they’re missing it’ll try to install them for you. It’ll also create a /nix directory if you didn’t have one and then see if there is an existing swap configured that should be kept when it switches to Nix.

Finally, the most important part, it’ll write the configuration files for your new NixOS instance based on the existing system’s configuration. This is where I got in to trouble. This function will create your configuration.nix, hardware-configuration.nix, and Optionally a networking.nix with the most basic configuration for NixOS, your current hardware, and your existing network configuration*.

With the configuration in place it’ll run infect() which installs/configures nix on the current system, builds your new NixOS configuration, and then does the dangerous part of deleting the current system and replacing it with the new NixOS one, including making changes to /boot. And then it’ll reboot the machine for you.

All of this is non-interactive so you can use this script directly within the cloud-init config of your favorite cloud or vps provider.

Everything goes wrong Link to heading

Before I ran nix-infect I checked my network configuration to see how it was I was getting an IP and what I might need to do to change the configuration so it continued to work. I was guessing it wasn’t complicated, but since nix-infect is meant to run on VMs I figured it was going to break for me. I noticed NIC was configured with a static IP configuration and when I checked the script I saw it could properly create the networking.nix file based off that configuration; the only condition was if [[ -n "$doNetConf" ]] && ... was true.

Easy.

However, I read that code as “if $doNetConf is not null run makeNetworkingConf(), but it did not. I cannot remember what I did wrong, all you need to do is set the variable to anything because -n in bash means a “non-zero length string”, but it didn’t create the networking configuration. And then it just went straight into rebooting so there wasn’t a way for me to check and notice that mistake.

When my machine rebooted it lost it’s public IP configuration. Making things more difficult, the iKVM for this instance didn’t work! Even if it had worked it probably wouldn’t have helped because nix-infect does not set a root password by default; it relies on copying the existing /root/.ssh/authorized_keys file from your existing install into your configuration so you can login which doesn’t help when I am trying to fix things via iKVM.

Getting NixOS installed… correctly this time Link to heading

Anyways, to fix this I fought with my host’s control panel to properly re-provision the machine with a working Debian 11 install, which I managed to get it to do eventually. Then I got to work debugging and modifying the script a bit.

First thing I did was make the script exit early before calling infect() so I could see the configuration it created. I was able to figure out that I’d misread how $doNetConf worked and it wasn’t creating a networking.nix.

I modified it to always generate the networking configuration and then modified the nix files some with some packages and configuration I wanted like enabling nix flakes. Then I edited the script again to comment out calling the function to generate the config and uncommented infect() to run the script again.

And… this time it worked!

Next steps Link to heading

My next steps are to write myself a nix flake for configuring the machine and setup deploy-rs so I can deploy changes to it that I built locally. That way I won’t be slowing down the machine when I am trying to do some kind of update if I don’t have to.

I also want to get this website building as a flake and somehow use that to deploy it to the machine and run it from there instead of Netlify.

Lots more to do to make proper use of that machine!