nixconfig/INSTALLATION.md

basic system installation

• The installations presented in this repository are always luks encrypted
• For simplicity I'm using device labels rather than uuids
• Secure Boot has to be disabled in the firmware, the installer iso isn't signed

1. create the partitions with cfdisk or fdisk (gpt)

   • nvme0n1p1: 2G, type EFI System, otherwise the firmware won't find the bootloader
   • nvme0n1p2: the rest, type Linux filesystem

   the layout should look somewhat like this after the installation

NAME               MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1            259:0    0 476.9G  0 disk
├─nvme0n1p1        259:1    0     2G  0 part  /boot
└─nvme0n1p2        259:2    0 474.9G  0 part
  └─cryptroot      254:0    0 474.9G  0 crypt
    ├─lvmroot-swap 254:1    0    20G  0 lvm   [SWAP]
    ├─lvmroot-home 254:2    0   250G  0 lvm   /home
    └─lvmroot-root 254:3    0 204.9G  0 lvm   /

Note: lsblk may additionally show /nix/store as a mountpoint on lvmroot-root. This is not a separate partition. NixOS mounts the root device a second time at /nix/store with ro,nosuid,nodev flags to enforce store immutability at runtime.

2. prepare the installation

# format the boot partition
mkfs.fat -F 32 /dev/nvme0n1p1 -n "nixboot"
# create an encrypted partition
cryptsetup luksFormat -y --label="nixcrypt" /dev/nvme0n1p2
# open the encrypted partition and map it to /dev/mapper/cryptroot
cryptsetup luksOpen /dev/nvme0n1p2 cryptroot

# create the physical volume
pvcreate /dev/mapper/cryptroot
# create a volume group inside
vgcreate lvmroot /dev/mapper/cryptroot
# create the swap volume, half the ram (no hibernation)
lvcreate --size 16G lvmroot --name swap
# if you desire, create a home volume
lvcreate --size 150G lvmroot --name home
# create the root volume, leave some space in the vg to grow volumes later
# (lvextend -r -L +50G lvmroot/home works while mounted)
lvcreate -l 90%FREE lvmroot --name root

# format as usual for root partition
mkfs.ext4 -L "nixroot" /dev/mapper/lvmroot-root
# if you previously made the home partition, do it too
mkfs.ext4 -L "nixhome" /dev/mapper/lvmroot-home
# format the swap partition
mkswap -L "nixswap" /dev/mapper/lvmroot-swap

# mount root
mount /dev/disk/by-label/nixroot /mnt
# mount boot
mount --mkdir /dev/disk/by-label/nixboot /mnt/boot
# again, if you did the home volume
mount --mkdir /dev/disk/by-label/nixhome /mnt/home
# turn on swap
swapon /dev/disk/by-label/nixswap

3. prepare nixos

# generate templates and update the hardware-configuration.nix
nixos-generate-config --root /mnt

# no extra kernelModules needed, luks and lvm load dm_crypt and dm_mod themselves

# add file systems using labels
fileSystems."/" =
  { device = "/dev/disk/by-label/nixroot";
    fsType = "ext4";
  };
fileSystems."/boot" =
  { device = "/dev/disk/by-label/nixboot";
    fsType = "vfat";
    options = [ "fmask=0077" "dmask=0077" ];  # only root may read /boot
  };
fileSystems."/home" =
  { device = "/dev/disk/by-label/nixhome";
    fsType = "ext4";
  };
swapDevices =
  [ { device = "/dev/disk/by-label/nixswap"; }
  ];

# point the bootloader to the luks device
boot.initrd.luks.devices."cryptroot" = {
  device = "/dev/disk/by-label/nixcrypt";
  # without this fstrim never reaches the ssd
  allowDiscards = true;
  # lower latency on nvme
  bypassWorkqueues = true;
};

4. install nixos

nixos-install

how to deploy the initial config

• Don't forget to install the bootloader, if you changed it since nixos-install
• The fresh install has no flakes and no git yet, a temporary shell is enough
• If the installer iso is older than the flake's nixpkgs, --no-reexec keeps nixos-rebuild from re-executing a newer version of itself, which can fail against the older systemd of the fresh install (only needed once)
• Set a password for the user before rebooting, users.nix doesn't set one

$ export NIX_CONFIG="experimental-features = nix-command flakes"
$ nix-shell -p git
$ sudo nixos-rebuild --install-bootloader boot --flake .#host_name --no-reexec
$ sudo passwd aaron

how to enable secure boot

Secure Boot uses lanzaboote with our own keys, see modules/nixos/lanzaboote.nix. Do this after the initial config is deployed and boots.

1. create the keys, before the first rebuild with the lanzaboote module

# the keys end up in /var/lib/sbctl, they never leave the host
$ sudo nix run nixpkgs#sbctl -- create-keys

2. import modules/nixos/lanzaboote.nix in the host's configuration.nix, rebuild and check the signatures

$ sudo nixos-rebuild switch --flake .#host_name
# EFI/Linux/*.efi and the bootloader must be signed, kernels in EFI/nixos
# are not, lanzaboote checks them by hash
$ sudo sbctl verify

3. reboot once with Secure Boot still disabled, to make sure the new entries boot

4. in the firmware (F2), erase all Secure Boot keys, then boot nixos again

# "Setup Mode: ✗ Enabled", the ✗ only means the state is insecure
$ sudo sbctl status

5. enroll the keys

# only our own keys, no microsoft keys
$ sudo sbctl enroll-keys

• sbctl refuses if the TPM eventlog shows option ROMs (e.g. a dedicated gpu). Don't force it, either trust exactly those with --tpm-eventlog or microsoft's keys with --microsoft. Without either, the firmware may not start the device, in the worst case the machine doesn't boot at all.
• Check what sbctl found first, a real option ROM has a PCI device path:

$ sudo nix shell nixpkgs#tpm2-tools -c tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements | grep -A14 "EV_EFI_BOOT_SERVICES_DRIVER"

• Paths starting with 0407 (firmware volume) are drivers from the firmware itself, not option ROMs (e.g. on the ThinkPad). --tpm-eventlog trusts exactly those, after a firmware update they may need to be enrolled again (disable Secure Boot, sbctl enroll-keys --tpm-eventlog)
• Without an integrated gpu (argon), the gpu's option ROM draws the firmware setup too, use --microsoft there, a refused gpu means no picture at all
• On the Framework (krypton), keep the firmware's built-in OEM certificates with --firmware-builtin, vendor firmware updates may need them. If an update fails or the keys were enrolled without them, erase the keys in the firmware again (Setup Mode) and re-enroll:

$ sudo sbctl enroll-keys --firmware-builtin

6. enable Secure Boot in the firmware and set a firmware password, then check

$ bootctl status | grep "Secure Boot"   # enabled (user)
$ sudo sbctl status                     # Secure Boot: ✓ Enabled

• If it doesn't boot anymore, disabling Secure Boot in the firmware boots as before
• Losing the keys isn't fatal: erase the keys in the firmware and start over at step 1

how to unlock the disk with the tpm (laptops)

Needs Secure Boot first. The TPM releases the key only with the PIN and while the Secure Boot state (PCR 7) is unchanged. The passphrase stays as fallback.

1. enroll the TPM with a PIN as an additional key slot

# asks for the current passphrase, then the new PIN
$ sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 --tpm2-with-pin=yes /dev/disk/by-label/nixcrypt

2. import modules/nixos/tpm-unlock.nix in the host's configuration.nix, rebuild and reboot

• If the passphrase is suddenly asked for instead of the PIN, the Secure Boot state changed. After a firmware update that's expected, otherwise investigate before typing the passphrase. To re-enroll:

$ sudo systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --tpm2-with-pin=yes /dev/disk/by-label/nixcrypt

how to upgrade the system

$ cd /path/to/repo
$ nix flake update
$ sudo nixos-rebuild switch --flake .#host_name
$ sudo nix-collect-garbage

how to use nix-helper

The tool nix-helper is installed by this configuration. It simplifies administrating nixos and adds more output to the rebuild command. It also features a diff after a successful build. The command uses the NH_FLAKE environment variable to be able to run from whatever directory.

Basic commands with a set NH_FLAKE variable are:

$ nh os switch
$ nh os build
$ nh os test
$ nh clean all --keep 5

There is also the option to interface with home-manager by using nh home switch but this isn't necessary since home-manager is imported as a module in this config.