nixconfig/INSTALLATION.md

# basic system installation

- The installations presented in this repository are always luks encrypted
- For simplicity I'm using device labels rather than uuids
- Secure Boot has to be disabled in the firmware, the installer iso isn't signed

1. create the partitions with `cfdisk` or `fdisk` (gpt)
   - `nvme0n1p1`: 2G, type `EFI System`, otherwise the firmware won't find the bootloader
   - `nvme0n1p2`: the rest, type `Linux filesystem`

   the layout should look somewhat like this after the installation
```bash
NAME               MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1            259:0    0 476.9G  0 disk
├─nvme0n1p1        259:1    0     2G  0 part  /boot
└─nvme0n1p2        259:2    0 474.9G  0 part
  └─cryptroot      254:0    0 474.9G  0 crypt
    ├─lvmroot-swap 254:1    0    20G  0 lvm   [SWAP]
    ├─lvmroot-home 254:2    0   250G  0 lvm   /home
    └─lvmroot-root 254:3    0 204.9G  0 lvm   /
```

> Note: `lsblk` may additionally show `/nix/store` as a mountpoint on `lvmroot-root`. This is not a separate partition. NixOS mounts the root device a second time at `/nix/store` with `ro,nosuid,nodev` flags to enforce store immutability at runtime.

2. prepare the installation
```bash
# format the boot partition
mkfs.fat -F 32 /dev/nvme0n1p1 -n "nixboot"
# create an encrypted partition
cryptsetup luksFormat -y --label="nixcrypt" /dev/nvme0n1p2
# open the encrypted partition and map it to /dev/mapper/cryptroot
cryptsetup luksOpen /dev/nvme0n1p2 cryptroot

# create the physical volume
pvcreate /dev/mapper/cryptroot
# create a volume group inside
vgcreate lvmroot /dev/mapper/cryptroot
# create the swap volume, half the ram (no hibernation)
lvcreate --size 16G lvmroot --name swap
# if you desire, create a home volume
lvcreate --size 150G lvmroot --name home
# create the root volume, leave some space in the vg to grow volumes later
# (lvextend -r -L +50G lvmroot/home works while mounted)
lvcreate -l 90%FREE lvmroot --name root

# format as usual for root partition
mkfs.ext4 -L "nixroot" /dev/mapper/lvmroot-root
# if you previously made the home partition, do it too
mkfs.ext4 -L "nixhome" /dev/mapper/lvmroot-home
# format the swap partition
mkswap -L "nixswap" /dev/mapper/lvmroot-swap

# mount root
mount /dev/disk/by-label/nixroot /mnt
# mount boot
mount --mkdir /dev/disk/by-label/nixboot /mnt/boot
# again, if you did the home volume
mount --mkdir /dev/disk/by-label/nixhome /mnt/home
# turn on swap
swapon /dev/disk/by-label/nixswap
```

3. prepare nixos
```bash
# generate templates and update the hardware-configuration.nix
nixos-generate-config --root /mnt

# no extra kernelModules needed, luks and lvm load dm_crypt and dm_mod themselves

# add file systems using labels
fileSystems."/" =
  { device = "/dev/disk/by-label/nixroot";
    fsType = "ext4";
  };
fileSystems."/boot" =
  { device = "/dev/disk/by-label/nixboot";
    fsType = "vfat";
    options = [ "fmask=0077" "dmask=0077" ];  # only root may read /boot
  };
fileSystems."/home" =
  { device = "/dev/disk/by-label/nixhome";
    fsType = "ext4";
  };
swapDevices =
  [ { device = "/dev/disk/by-label/nixswap"; }
  ];

# point the bootloader to the luks device
boot.initrd.luks.devices."cryptroot" = {
  device = "/dev/disk/by-label/nixcrypt";
  # without this fstrim never reaches the ssd
  allowDiscards = true;
  # lower latency on nvme
  bypassWorkqueues = true;
};
```

4. install nixos
```bash
nixos-install
```

## how to deploy the initial config
- Don't forget to install the bootloader, if you changed it since `nixos-install`
- The fresh install has no flakes and no git yet, a temporary shell is enough
- If the installer iso is older than the flake's nixpkgs, `--no-reexec` keeps
  `nixos-rebuild` from re-executing a newer version of itself, which can fail
  against the older systemd of the fresh install (only needed once)
- Set a password for the user before rebooting, `users.nix` doesn't set one
```bash
$ export NIX_CONFIG="experimental-features = nix-command flakes"
$ nix-shell -p git
$ sudo nixos-rebuild --install-bootloader boot --flake .#host_name --no-reexec
$ sudo passwd aaron
```

## how to enable secure boot
Secure Boot uses lanzaboote with our own keys, see `modules/nixos/lanzaboote.nix`.
Do this after the initial config is deployed and boots.

1. create the keys, before the first rebuild with the lanzaboote module
```bash
# the keys end up in /var/lib/sbctl, they never leave the host
$ sudo nix run nixpkgs#sbctl -- create-keys
```

2. import `modules/nixos/lanzaboote.nix` in the host's `configuration.nix`, rebuild and check the signatures
```bash
$ sudo nixos-rebuild switch --flake .#host_name
# EFI/Linux/*.efi and the bootloader must be signed, kernels in EFI/nixos
# are not, lanzaboote checks them by hash
$ sudo sbctl verify
```

3. reboot once with Secure Boot still disabled, to make sure the new entries boot

4. in the firmware (F2), erase all Secure Boot keys, then boot nixos again
```bash
# "Setup Mode: ✗ Enabled", the ✗ only means the state is insecure
$ sudo sbctl status
```

5. enroll the keys
```bash
# only our own keys, no microsoft keys
$ sudo sbctl enroll-keys
```
- sbctl refuses if the TPM eventlog shows option ROMs (e.g. a dedicated gpu).
  Don't force it, either trust exactly those with `--tpm-eventlog` or
  microsoft's keys with `--microsoft`. Without either, the firmware may not
  start the device, in the worst case the machine doesn't boot at all.
- Check what sbctl found first, a real option ROM has a PCI device path:
```bash
$ sudo nix shell nixpkgs#tpm2-tools -c tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements | grep -A14 "EV_EFI_BOOT_SERVICES_DRIVER"
```
- Paths starting with `0407` (firmware volume) are drivers from the firmware
  itself, not option ROMs (e.g. on the ThinkPad). `--tpm-eventlog` trusts
  exactly those, after a firmware update they may need to be enrolled again
  (disable Secure Boot, `sbctl enroll-keys --tpm-eventlog`)
- Without an integrated gpu (argon), the gpu's option ROM draws the firmware
  setup too, use `--microsoft` there, a refused gpu means no picture at all

6. enable Secure Boot in the firmware and set a firmware password, then check
```bash
$ bootctl status | grep "Secure Boot"   # enabled (user)
$ sudo sbctl status                     # Secure Boot: ✓ Enabled
```
- If it doesn't boot anymore, disabling Secure Boot in the firmware boots as before
- Losing the keys isn't fatal: erase the keys in the firmware and start over at step 1

## how to unlock the disk with the tpm (laptops)
Needs Secure Boot first. The TPM releases the key only with the PIN and while
the Secure Boot state (PCR 7) is unchanged. The passphrase stays as fallback.

1. enroll the TPM with a PIN as an additional key slot
```bash
# asks for the current passphrase, then the new PIN
$ sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 --tpm2-with-pin=yes /dev/disk/by-label/nixcrypt
```

2. import `modules/nixos/tpm-unlock.nix` in the host's `configuration.nix`, rebuild and reboot

- If the passphrase is suddenly asked for instead of the PIN, the Secure Boot
  state changed. After a firmware update that's expected, otherwise investigate
  before typing the passphrase. To re-enroll:
```bash
$ sudo systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --tpm2-with-pin=yes /dev/disk/by-label/nixcrypt
```

## how to upgrade the system
```bash
$ cd /path/to/repo
$ nix flake update
$ sudo nixos-rebuild switch --flake .#host_name
$ sudo nix-collect-garbage
```

## how to use nix-helper

The tool nix-helper is installed by this configuration. It simplifies administrating nixos and adds more output to the rebuild command. It also features a diff after a successful build. The command uses the `NH_FLAKE` environment variable to be able to run from whatever directory.

Basic commands with a set `NH_FLAKE` variable are:
```bash
$ nh os switch
$ nh os build
$ nh os test
$ nh clean all --keep 5
```

There is also the option to interface with home-manager by using `nh home switch` but this isn't necessary since home-manager is imported as a module in this config.