Merge pull request '26-install-zfs-only-when-needed' (#27) from 26-install-zfs-only-when-needed into main

Reviewed-on: #27

.gitignore

@@ -0,0 +1,2 @@
+archzbm_settings.env
+.idea

README.md

@@ -6,56 +6,134 @@ Helper script to install Arch Linux with ZFSBootMenu from within a running Arch
 
 We expect minimal prep on your end. Please make sure that before execution the following conditions are met.
 
+### UEFI
+
+On a UEFI system ensure these conditions are met. See [How to prep](#how-to-prep) for details on how to meet these conditions.
+
+- One GPT-partitioned disk
 - Arch Linux live CD ISO image sees exactly one partition with partition type code `BF00` ("Solaris root")
-- Arch Linux live CD ISO image sees exactly one partition with partition type code `EF00` ("EFI system partition")
+- Arch Linux live CD ISO image sees exactly one partition with partition type code `EF00` ("EFI System Partition")
 - The `EF00` EFI partition is mountable, in practical terms this usually only means it has a file system.
 - No ZFS zpool exists
 
-### How to prep
+### Legacy BIOS
 
-On a blank example disk `/dev/sda` you can fulfill the requirements (One `EF00` partition with a file system plus one `BF00` partition) for example like so:
+If you are instead running a legacy BIOS machine ensure these conditions are met. See [How to prep](#how-to-prep) for details on how to meet these conditions.
+
+- One MBR-partitioned disk
+- Arch Linux live CD ISO image sees exactly one partition with partition type code `bf` ("Solaris root")
+- Arch Linux live CD ISO image sees exactly one partition with partition type code `83` ("Linux")
+- The `83` Linux partition is mountable, in practical terms this usually only means it has a file system.
+- No ZFS zpool exists
+
+Neither with a UEFI nor legacy BIOS system are any of these conditions a requirement from ZFSBootMenu. We're just setting requirements to easily identify if you intend to do a UEFI or a legacy BIOS install. Subsequently the script has no logic to detect UEFI or legacy BIOS mode, that's legwork left to the reader :) The Internet seems to agree that a good quick check is to see if your Arch Linux live CD ISO image has directory `/sys/firmware/efi`.
 ```
-sgdisk --new '1::+100M' --new '2' --typecode '1:EF00' --typecode '2:BF00' /dev/sda
+[ -d /sys/firmware/efi ] && echo 'Likely a UEFI system' || echo 'Probably a legacy BIOS system'
+```
+If you're unsure nothing's stopping you from just giving it a go with a best guess and if that fails you know you guessed wrong.
+
+## How to prep
+
+### UEFI
+
+On a blank example disk `/dev/sda` you can fulfill the UEFI requirements (one `EF00` partition with a file system plus one `BF00` partition) for example like so:
+```
+sgdisk --new '1::+512M' --new '2' --typecode '1:EF00' --typecode '2:BF00' /dev/sda
 mkfs.vfat /dev/sda1
 ```
-> `--new '1::+100M'`: Create partition number `1`. The field separator `:` separates the partition number from start sector. In this case start sector is unspecified so start sector sits at whatever the system's default is for this operation. On a blank disk on an Arch Linux live CD ISO image this will default to sector `2048`. Partition ends at whatever the beginning is `+100M` meaning plus 100 Mebibytes.
+> `--new '1::+512M'`: Create partition number `1`. The field separator `:` separates the partition number from start sector. In this case start sector is unspecified so start sector sits at whatever the system's default is for this operation. On a blank disk on an Arch Linux live CD ISO image this will default to sector `2048`. Partition ends at whatever the beginning is `+512M` meaning plus 512 Mebibytes.
 >
 > `--new '2'`: Create partition number `2`. Both field number 2, the start sector, and field number 3, the end sector, are unspecified, there's no field separator `:`. Field number 2 will be the first free sector - in this case right after partition 1 - and field number 3 will be end of disk. Thus partition `2` will fill the remaining free disk space.
 >
-> `--typecode '1:EF00'`: Partition 1 gets partition type code `EF00`, an EFI system partition.
+> `--typecode '1:EF00'`: Partition 1 gets partition type code `EF00`, an EFI System Partition.
 >
 > `--typecode '2:BF00'`: Partition 2 gets partition type code `BF00`, a Solaris root partition.
 
 The result will be something like this at which point you can start the `setup.sh` script, see [How to run this?](#how-to-run-this) below for more details.
 ```
-# lsblk --paths 
-NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
-/dev/loop0     7:0    0 688.5M  1 loop /run/archiso/airootfs
-/dev/sr0      11:0    1 810.3M  0 rom  /run/archiso/bootmnt
-/dev/sda     202:0    0    10G  0 disk 
-├─/dev/sda1  202:1    0   100M  0 part
-└─/dev/sda2  202:2    0   9.9G  0 part 
+# lsblk --paths --output 'NAME,SIZE,FSTYPE,PARTTYPE,PARTTYPENAME,PTTYPE' /dev/sda
+NAME         SIZE FSTYPE PARTTYPE                             PARTTYPENAME PTTYPE
+/dev/sda      10G                                                          gpt
+├─/dev/sda1  512M vfat   c12a7328-f81f-11d2-ba4b-00a0c93ec93b EFI System   gpt
+└─/dev/sda2  9.5G        6a85cf4d-1dd2-11b2-99a6-080020736631 Solaris root gpt
 ```
 
-## ZFS dataset layout
+### Legacy BIOS
 
-The script will create a single ZFS zpool `zpool` on the `BF00` partition with dataset child `zpool/root` which itself has one child `zpool/root/archlinux`, that's where Arch Linux gets installed. Parallel to `zpool/root` it'll create `zpool/data` with a `zpool/data/home` child dataset that gets mounted at `/home`.
+For a legacy BIOS machine you'll be using a Master Boot Record (MBR) on your disk.
+```
+printf -- '%s\n' 'label: dos' 'start=1MiB, size=512MiB, type=83, bootable' 'start=513MiB, size=+, type=bf' | sfdisk /dev/sda
+mkfs.vfat /dev/sda1
+```
+> `label: dos`: Create the following partition layout in a Master Boot Record.
+>
+> `start=1MiB, size=512MiB, type=83, bootable`: Partition 1 begins 1 Mebibyte after disk start and is 512 Mebibyte in size. We're setting its bootable flag and setting partition type code `83` ("Linux").
+>
+> `start=513MiB, size=+, type=bf`: Partition 2 begins right at the start of Mebibyte 513, this is the very next sector after the end of partition 1. It takes up the remaining disk space, we're assigning type code `bf` ("Solaris").
 
-The script will use the `EF00` partition to install a ZFSBootMenu EFI executable if `efibootmgr` says that no such `ZFSBootMenu` entry exists. If ZFSBootMenu gets added to the EFI partition it'll become primary boot option.
+The result will be something like this at which point you can start the `setup.sh` script, see [How to run this?](#how-to-run-this) below for more details.
+```
+# lsblk --paths --output 'NAME,SIZE,FSTYPE,PARTTYPE,PARTTYPENAME,PTTYPE' /dev/sda
+NAME         SIZE FSTYPE PARTTYPE PARTTYPENAME PTTYPE
+/dev/sda      10G                              dos
+├─/dev/sda1  512M vfat   0x83     Linux        dos
+└─/dev/sda2  9.5G        0xbf     Solaris      dos
+```
 
-## How to run this?
+# Partition naming
+
+Since this script works with UEFI and legacy BIOS mode we'll be addressing both disk layout schemes with umbrella terms for the rest of this document for better readability: "The zpool partition" will be GPT `BF00` partition and MBR `bf` partition. You'll parse the text accordingly. "The boot partition" will be GPT `EF00` partition as well as the MBR `83` partition.
+
+# ZFS dataset layout
+
+The script will create a single ZFS zpool `zpool` on the zpool partition with dataset child `zpool/root` which itself has one child `zpool/root/archlinux`, that's where Arch Linux gets installed. Parallel to `zpool/root` it'll create `zpool/data` with a `zpool/data/home` child dataset that gets mounted at `/home`.
+
+# How to run this?
 
 - Boot an Arch Linux live CD ISO image
 - Run:
     ```
-    export SCRIPT_URL='https://quico.space/quico-os-setup/arch-zbm/raw/branch/main/setup.sh'
-    curl -s "${SCRIPT_URL}" | bash
+    export SCRIPT_URL='https://quico.space/quico-os-setup/arch-zbm/raw/branch/main/setup.sh' && curl -s "${SCRIPT_URL}" | bash
     ```
     During execution the script will call itself when it changes into its `chroot`, that's why we `export SCRIPT_URL`. Feel free to update `"${SCRIPT_URL}"` with whatever branch or revision you want to use from [quico.space/quico-os-setup/arch-zbm](https://quico.space/quico-os-setup/arch-zbm). Typically `.../branch/main/setup.sh` as shown above is what you want.
 
-### Options
+## Options
 
-#### Compression
+The following options can be given either by exporting them as shell variables prior to script execution or in a file named `archzbm_settings.env` that lives in your current working directory where you're about to execute the script. You can walk yourself through an interactive questionnaire that helps create a valid `archzbm_settings.env` file. Check out [Command line setup help](#command-line-setup-help) for details on the questionnaire.
+
+If you instead want to define settings yourself with an `archzbm_settings.env` file its file format is identical to shell variable assignments of the form `VAR=value` or `VAR='value'`.
+
+If `./archzbm_settings.env` exists the script will `source` its content and `export` all variables for use in future steps.
+
+In cases where a variable is both exported prior to script execution and specified in `archzbm_settings.env` the latter will override the former.
+
+Known options are as follows.
+
+### Kernel downgrade
+
+By default we install newest `linux` and `linux-headers` packages into a `chroot`. Once we're in that `chroot` we then install newest [AUR zfs-dkms package](https://aur.archlinux.org/packages/zfs-dkms). You may want to override `linux` and `linux-headers` versions to ensure you end up with a compatible mix between them and `zfs-dkms`.
+
+For example:
+```
+export ARCHZBM_KERNEL_VER=6.5.9.arch2
+```
+
+In our `chroot` this will trigger execution of:
+```
+downgrade --ala-only 'linux=6.5.9.arch2' 'linux-headers=6.5.9.arch2' --ignore always
+```
+
+Where `downgrade` is the [AUR downgrade package](https://aur.archlinux.org/packages/downgrade). This will downgrade `linux` and `linux-headers` and will add a setting to your `/etc/pacman.conf`:
+```
+[options]
+IgnorePkg = linux linux-headers
+```
+
+Setting `ARCHZBM_KERNEL_VER` to an empty string `''` or keeping it undefined are both valid and will retain newest versions instead of downgrading.
+
+Also read [Kernel selection](#kernel-selection) for details.
+
+### Compression
 
 By default we create a zpool with ZFS property `compression=on`. If the `lz4_compress` pool feature is active this will by default enable `compression=lz4`. See `man 7 zfsprops` for example in ZFS 2.1.9 for details. See `zpool get feature@lz4_compress <pool>` to check this feature's status on your `<pool>`.
 
@@ -64,7 +142,7 @@ To get a zpool with uncompressed datasets export the shell variable `ARCHZBM_ZFS
 export ARCHZBM_ZFSPROPS_NO_COMPRESSION=yesplease
 ```
 
-#### Encryption
+### Encryption
 
 By default we encrypt the zpool with ZFS property `encryption=on`. In ZFS 2.1.9 this defaults to `encryption=aes-256-gcm`.
 
@@ -73,19 +151,194 @@ To get a zpool with unencrypted datasets export the shell variable `ARCHZBM_ZFSP
 export ARCHZBM_ZFSPROPS_NO_ENCRYPTION=yup
 ```
 
-## Steps
+### Passwords
+
+By default both the zpool password and the account password for `root` are literally `password`. While you can certainly change these after initial system setup (see [Password change](#password-change)) you can also optionally set passwords before script execution as follows:
+
+```
+ARCHZBM_ZPOOL_PASSWORD='a fancy password'
+ARCHZBM_ROOT_PASSWORD='t0psecr3t!'
+```
+
+> While the `root` password is allowed to be weak and `chpasswd` won't care do make sure to set a zpool password that meets ZFS' complexity rules. Per `man 7 zfsprops` section `keyformat` the only requirement is a length "between 8 and 512 bytes" (as in minimum 8 characters). If you pick a password that's too weak ZFS will reject zpool creation and very ungracefully derail the rest of this script. The script doesn't check what you're setting.
+
+The script does create a second user named `build` but doesn't set a password on account creation. It's intended as a helper for system setup tasks such as `sudo -u build paru -S <package>` where an account password is irrelevant since `root` can always `sudo` whatever it wants. You will not be able to log in to the `build` account yourself although you certainly could set a password for it. Instead we suggest you create a proper user account for yourself. Your newly installed Arch Linux comes with an `/etc/motd` greeting that summarizes this as:
+
+```
+useradd --create-home --shell /bin/bash --user-group --groups wheel <user>
+passwd <user>
+```
+
+### Networking
+
+By default the script configures plain ZFSBootMenu without networking nor an SSH server. If you're interested in SSH-ing into your ZFSBootMenu boot loader you're going to want to specify some of the following variables.
+
+#### IP address
+
+> IPv6 addresses are untested. Script has been confirmed working with IPv4 addresses.
+
+```
+ARCHZBM_NET_CLIENT_IP=''
+ARCHZBM_NET_SERVER_IP=''
+ARCHZBM_NET_GATEWAY_IP=''
+ARCHZBM_NET_NETMASK=''
+ARCHZBM_NET_HOSTNAME=''
+ARCHZBM_NET_DEVICE=''
+ARCHZBM_NET_AUTOCONF=''
+```
+
+By default none of the variables are set to any value and no networking will be available in ZFSBootMenu. If you want networking as in an IP address bound to a network interface set at least one of these variables or one of the [SSH](#ssh) variables listed further down. Setting one or more `ARCHZBM_NET_*` variables to an empty string is valid. If at least one variable is given either from this paragraph or from [SSH](#ssh) we're assuming that you want networking. Unspecified values and values set to the empty string `''` use defaults.
+
+For networking we rely on the [mkinitcpio-nfs-utils](https://archlinux.org/packages/core/x86_64/mkinitcpio-nfs-utils/) package with its `net` hook. Please refer to its [initcpio-install-net](https://gitlab.archlinux.org/archlinux/packaging/packages/mkinitcpio-nfs-utils/-/blob/main/initcpio-install-net) script file for usage hints on above variables. The hook implements a subset of the [ip Kernel Command Line argument](https://docs.kernel.org/admin-guide/nfs/nfsroot.html).
+
+Mapping between `net` hook field names and our shell variables is straightforward. Fields 8, 9 and 10 (DNS and NTP server addresses) from the official `ip` docs are unsupported in `net` hook. As such our hook has a total of 7 fields available for you to configure.
+
+```
++-------------+------------------------+
+|  net hook   |       This script      |
++-------------+------------------------+
+| <client-ip> | ARCHZBM_NET_CLIENT_IP  |
+| <server-ip> | ARCHZBM_NET_SERVER_IP  |
+| <gw-ip>     | ARCHZBM_NET_GATEWAY_IP |
+| <netmask>   | ARCHZBM_NET_NETMASK    |
+| <hostname>  | ARCHZBM_NET_HOSTNAME   |
+| <device>    | ARCHZBM_NET_DEVICE     |
+| <autoconf>  | ARCHZBM_NET_AUTOCONF   |
++-------------+------------------------+
+```
+
+A valid example with a few fields populated may look like so:
+
+```
+ARCHZBM_NET_CLIENT_IP='10.10.10.2'
+ARCHZBM_NET_GATEWAY_IP='10.10.10.1'
+ARCHZBM_NET_NETMASK='255.255.255.0'
+ARCHZBM_NET_DEVICE='eth0'
+ARCHZBM_NET_AUTOCONF='none'
+```
+
+Note that in this example `ARCHZBM_NET_SERVER_IP` and `ARCHZBM_NET_HOSTNAME` are left unassigned.
+
+It'll add the following `ip=` instruction to your Kernel Command Line:
+```
+ip=10.10.10.2::10.10.10.1:255.255.255.0::eth0:none
+```
+
+This is also valid and will configure `eth0` via DHCP:
+
+```
+ARCHZBM_NET_DEVICE='eth0'
+ARCHZBM_NET_AUTOCONF='dhcp'
+```
+
+> In ZFSBootMenu the device names that go into `ARCHZBM_NET_DEVICE` are raw unchanged kernel device names such as `eth0`. If you're unsure which device name to use in your Arch Linux live CD ISO image check `dmesg` output. During boot typically a kernel module will first assign the raw kernel device name then later `systemd` will enforce [Predictable Network Interface Names](https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/).
+>
+> In `dmesg | grep` on a physical PC with an MSI B550-A Pro mainboard from 2020 that comes with one onboard Realtek RTL8111H network adapter governed by the Realtek RTL-8169 Gigabit Ethernet driver from the `r8169` kernel module you will for example see:
+> ```
+> # dmesg -T | grep eth
+> [time] r8169 0000:2a:00.0 eth0: RTL8168h/8111h, 04:7c:16:00:01:02, XID 541, IRQ 95
+> [time] r8169 0000:2a:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
+> [time] r8169 0000:2a:00.0 enp42s0: renamed from eth0
+> ```
+>
+> Notice how a Predictable Network Interface Name comes in on line 3. What *__you__* need here is the `eth0` part.
+
+#### SSH
+
+If you want networking indicated by the fact that at least one of the `ARCHZBM_NET_*` variables is set or one of the `ARCHZBM_SSH_*` vars we assume that you want an SSH daemon as well. This comes in the form of a `dropbear` daemon with minimal configurability. Use the following variables to define Dropbear's behavior.
+
+```
+ARCHZBM_SSH_PORT='22'
+ARCHZBM_SSH_KEEPALIVE_INTVL='1'
+ARCHZBM_SSH_AUTH_KEYS=''
+```
+
+In `ARCHZBM_SSH_PORT` you specify Dropbear's listening port, this defaults to `22` if unconfigured or set to an empty string. With `ARCHZBM_SSH_KEEPALIVE_INTVL` you define at which interval Dropbear will send keepalive messages to an SSH client through the SSH connection. This defaults to `1` as in every `1` second a keepalive message is sent. Per [man 8 dropbear](https://man.archlinux.org/man/extra/dropbear/dropbear.8.en) a value of `0` disables Dropbear sending keepalive messages. We suggest to leave this on and to keep the interval short, see [SSH in ZFSBootMenu](#ssh-in-zfsbootmenu) for how to work with this.
+
+Dropbear in this setup only supports key-based authentication, no password-based authentication. The value from `ARCHZBM_SSH_AUTH_KEYS` will be converted to a list of public SSH keys allowed to SSH into Dropbear as its default `root` user while ZFSBootMenu is running. The format of `ARCHZBM_SSH_AUTH_KEYS` is a single line where `authorized_keys` entries are split with double-commas:
+
+```
+ssh-rsa Eahajei8,,ssh-ed25519 kaeD0mas ...
+```
+
+This syntax crutch allows you to use the full range of Dropbear-supported `authorized_keys` stanzas, see [man 8 dropbear](https://man.archlinux.org/man/extra/dropbear/dropbear.8.en) for what's available. Whether or not this is useful to you is another topic :) At least the functionality for stanzas is there by separating values in `ARCHZBM_SSH_AUTH_KEYS` with double-commas.
+
+## Command line setup help
+
+An interactive questionnaire can guide you through settings and goes like this:
+
+![Command line setup questionnaire](https://i.imgur.com/OXG75GH.gif)
+
+To do the questionnaire yourself start this script with the `setup` argument:
+
+```
+export SCRIPT_URL='https://quico.space/quico-os-setup/arch-zbm/raw/branch/main/setup.sh' && curl -s "${SCRIPT_URL}" | bash -s -- setup
+```
+
+When done rerun it without that argument:
+
+```
+export SCRIPT_URL='https://quico.space/quico-os-setup/arch-zbm/raw/branch/main/setup.sh' && curl -s "${SCRIPT_URL}" | bash
+```
+
+# Steps
 
 The script takes the following installation steps.
 
 1. Install ZFS tools and kernel module with [github.com/eoli3n/archiso-zfs](https://github.com/eoli3n/archiso-zfs)
-1. Create one ZFS zpool on top of `BF00` partition, encrypted and compressed datasets, password `password`
+1. Create one ZFS zpool on top of zpool partition, encrypted and compressed datasets, password `password`
+    1. _See paragraph [Passwords](#passwords) to predefine your own passwords in a settings file_
     1. _See paragraphs [Compression](#compression)/[Encryption](#encryption) to optionally disable properties_
 1. Create dataset for Arch Linux and `/home`
 1. Install Arch Linux into pool
-1. Add ZFSBootMenu to `EF00` partition if it doesn't exist already
+1. Add ZFSBootMenu to boot partition
+1. Configure boot method
+    - Either an EFI image with EFI boot order entries on a UEFI machine
+    - Or Syslinux with `extlinux` for a legacy BIOS computer
+1. If requested by user enable SSH in ZFSBootMenu. We then also add:
+    - [quico.space/quico-os-setup/mkinitcpio-dropbear-pacman-hook](https://quico.space/quico-os-setup/mkinitcpio-dropbear-pacman-hook)
+1. Add `pacman` hooks to keep ZFSBootMenu images (and `extlinux`) updated
+    - [quico.space/quico-os-setup/zbm-regen-pacman-hook](https://quico.space/quico-os-setup/zbm-regen-pacman-hook)
+    - [quico.space/quico-os-setup/zbm-syslinux-pacman-hook](https://quico.space/quico-os-setup/zbm-syslinux-pacman-hook)
 1. Exit into Arch Linux live CD ISO image shell for you to `reboot` and frolick
 
-## Flavor choices
+# SSH in ZFSBootMenu
+
+Per [SSH](#ssh) and [Networking](#networking) this script will optionally add a Dropbear SSH daemon to ZFSBootMenu. While the mechanism of SSH-ing into a server isn't particularly noteworthy we humbly suggest that in this particular use case you let your SSH client listen for keepalive messages from the server.
+
+```
+ssh -o ServerAliveInterval=3 -o ServerAliveCountMax=0 root@<addr> -p <port>
+```
+
+A typical workflow with Dropbear is for you to SSH into it, issue `zfs` or `zfsbootmenu` commands and allow the Arch Linux boot process to commence. As soon as you're done Dropbear will terminate as ZFSBootMenu hands control off to your operating system's kernel. Without your client listening to keepalive messages it may not realize that the connection's gone for quite some time until you harshly interrupt it.
+
+The server defaults to sending keepalive messages to your client every second.
+
+With `-o ServerAliveInterval=3` you instruct your client to send an are-your-still-there message to the server if your client ever stops getting keepalive messages from the server for 3 seconds. The server defaults to sending 1 keepalive ping per second so even on a somewhat lossy connection we can reasonably expect to get one message through to us within 3 seconds.
+
+When it comes to the point that your SSH client sends an are-your-still-there message it expects a near-realtime response. It will accept `-o ServerAliveCountMax=0` failures from the server to comply.
+
+This effectively configures your SSH client to remain connected even through somewhat lossy hops to the Dropbear daemon; and to cleanly disconnect 3 seconds and some change after you've executed whatever you needed to do in ZFSBootMenu.
+
+# Kernel selection
+
+This script compiles ZFS via Arch Linux' [Dynamic Kernel Module Support](https://wiki.archlinux.org/title/Dynamic_Kernel_Module_Support) (DKMS). Not all kernels allow for successful compilation, in some instances a particularly recent kernel version may change APIs to such a degree that ZFS compilation simply fails.
+
+We strongly suggest to that you:
+
+- Firstly, refer to a resource such as the [Arch Linux Archive package version list](https://archive.archlinux.org/packages/l/linux/) to find out what newest kernel version this script will install.
+- Secondly, research if newest [AUR zfs-dkms package](https://aur.archlinux.org/packages/zfs-dkms) is compatible with that kernel. Two reasonable points of contact are AUR and its comments section for `zfs-dkms` where users quickly report issues; and the [github.com/openzfs/zfs issues list](https://github.com/openzfs/zfs/issues).
+
+An example for this is that `linux-6.6.1.arch1-1-x86_64` came out on Wednesday, November 8, 2023 at a time when newest `zfs-dkms` package version [was 2.2.0](https://aur.archlinux.org/cgit/aur.git/commit/?h=zfs-dkms&id=da1b6372c57b16f2781a7fda2b95971bb392c5ee) which did not compile against `linux` 6.6.x.
+
+You'd then set for example:
+```
+export ARCHZBM_KERNEL_VER=6.5.9.arch2
+```
+
+Where any 6.5.x version is known to work well with `zfs-dkms`. See also [Kernel downgrade](#kernel-downgrade) for details on how to configure this.
+
+# Flavor choices
 
 We make the following opinionated flavor choices. Feel free to change them to your liking.
 
@@ -97,19 +350,19 @@ We make the following opinionated flavor choices. Feel free to change them to yo
 - Timezone is `Etc/UTC`
     - Check `timedatectl set-timezone <tzdata-zone>`
 
-## Post-run manual steps
+# Post-run manual steps
 
 After installation you're going to want to at least touch these points in your new Arch Linux install:
 
 - Package manager hook: `pacman` does not have a hook to do ZFS snapshots
-    - See [this GitHub gist](https://gist.github.com/Soulsuke/6a7d1f09f7fef968a2f32e0ff32a5c4c#file-arch_on_zfs-txt-L238) and [zfs-snapshotter.bash](https://github.com/Soulsuke/arch-zfs-tools/blob/master/zfs-snapshotter.bash) for inspiration
+    - See [quico.space/quico-os-setup/zfs-pacman-hook](https://quico.space/quico-os-setup/zfs-pacman-hook/src/branch/1-get-base-version-going) for an example you may want to install
 - Hostname: Installation chose a pseudo-randomly generated 8-character string with `pwgen`
     - Check `hostnamectl set-hostname <hostname>`
 - Unprivileged user accounts: The OS was installed with `root` and unprivileged `build` users
-- Passwords
+- Unless you had a settings file or exported shell env vars per [Passwords](#passwords) you're going to want to change passwords now:
     - ZFS: The password for all datasets underneath `zpool` is `password`.
     - Local `root` account: The local `root` account's password is `password`.
-- Arch User Repository (AUR) helper: We installed [paru](https://github.com/Morganamilo/paru) as our AUR helper, we installed from GitHub via `makepkg -si`.
+- Arch User Repository (AUR) helper: We installed [paru](https://github.com/Morganamilo/paru) as our AUR helper, we installed from AUR as [paru-bin](https://aur.archlinux.org/packages/paru-bin).
 - In `/etc/systemd/network/50-wired.network` instead of a DHCP-based network config you can get a static one. The DHCP-based one for reference looks like:
     ```
     ...
@@ -134,10 +387,16 @@ After installation you're going to want to at least touch these points in your n
     IPForward=yes
     Domains=~.
     ```
+- In case you later want a graphical interface and specifically NetworkManager (via package `networkmanager`) consider telling it to keep its hands off of some of your network interfaces. The bullet point above adds a `systemd`-style config file that `systemd-networkd.service` will read and use. Should you ever install NetworkManager it will by default assume that it must manage all interfaces. It'll use its own DHCP client to try and get IP addresses for _managed interfaces_ in which case you'll end up with whatever addressing scheme you configured in a `.network` unit file plus NetworkManager's additional address. Create `/etc/NetworkManager/conf.d/99-unmanaged-devices.conf` for example to declare some interfaces as off-limits or _unmanaged_:
+    ```
+    [keyfile]
+    unmanaged-devices=mac:52:54:00:74:79:56;type:ethernet
+    ```
+    Check out [ArchWiki article "NetworkManager" section "Ignore specific devices"](https://wiki.archlinux.org/title/NetworkManager#Ignore_specific_devices) for more info.
 
 # Password change
 
-After installation you're going to want to change your ZFS encryption password.
+After installation you're going to want to change your ZFS encryption password (unless you preconfigured a good zpool password in a settings file per [Passwords](#passwords)). At any rate you still want to be familiar with the process and its caveat in case you ever need a zpool password change or want to do one now.
 
 ## Steps
 
@@ -173,7 +432,7 @@ ZFS generates the master key exactly once when you enable encryption on a datase
 
 `man 8 zfs-change-key` from `zfs-utils` version 2.1.9 adds:
 > If the user's key is compromised, `zfs change-key` does not necessarily protect existing or newly-written data from attack. Newly-written data will continue to be encrypted with the same master key as the existing data. The master key is compromised if an attacker obtains a user key and the corresponding wrapped master key. Currently, `zfs change-key` does not overwrite the previous wrapped master key on disk, so it is accessible via forensic analysis for an indeterminate length of time.
-> 
+>
 > In the event of a master key compromise, ideally the drives should be securely erased to remove all the old data (which is readable using the compromised master key), a new pool created, and the data copied back. This can be approximated in place by creating new datasets, copying the data (e.g. using `zfs send | zfs recv`), and then clearing the free space with `zpool trim --secure` if supported by your hardware, otherwise `zpool initialize`.
 
 On one hand changing the ZFS encryption password is generally a good and useful thing to do. On the other hand changing your password does not currently overwrite previous wrapped master keys on disk. A sufficiently motivated party that gains access to a wrapped master key and the matching user key is able to decrypt the master key and use it to read all data encrypted with it.
@@ -201,7 +460,7 @@ In order to generate a new master key after you've changed your user key as ment
         --large-block \
         --compressed \
         'zpool/root/archlinux-sxu@rekey' | \
-
+    \
     zfs receive \
         -Fvu \
         -o 'encryption=on' \
@@ -291,6 +550,7 @@ The ZFS pool and dataset setup that makes this tick, explained in plain English.
     1. `-O keylocation=file://...`: This property is only set for encrypted datasets which are encryption roots. Controls where the user's encryption key will be loaded from by default for commands such as `zfs load-key`.
     1. `-O keyformat=passphrase`: Controls what format the user's encryption key will be provided as. Passphrases must be between 8 and 512 bytes long.
 1. At this time the newly created zpool is not mounted anywhere. Next we create the "root" dataset, that's an arbitary term for the parent dataset of all boot environments. Boot environments in your case may be for example different operating systems all of which live on separate datasets underneath the root.
+    1. `-o canmount=off`: Same as above, the root dataset can - just like the pool - not be mounted.
     1. `-o mountpoint=none`: Same as above, the root dataset has - just like the pool - no mountpoint configured.
     1. `zfs set org.zfsbootmenu:commandline=...`: Set a common kernel command line for all boot environments such as `"ro quiet"`.
 1. Neither the root dataset nor the pool are mounted at this time. We now create one boot environment dataset where we want to install Arch Linux.
@@ -308,7 +568,7 @@ The ZFS pool and dataset setup that makes this tick, explained in plain English.
 1. We export the zpool once, we then reimport it by scanning only inside `/dev/disk/by-partuuid`, again setting `-R /mnt` as we did during pool creation a moment ago and we do not mount any file systems.
 1. We `zfs load-key <encryptionroot>` which will load the key from `keylocation` after which the `keystatus` property for `<encryptionroot>` and all child datasets will change from `unavailable` to `available`.
 1. We mount our Arch Linux boot environment dataset. It automatically gets prefixed with `-R /mnt` since that's how we imported the pool.
-1. We `zfs mount -a` which automounts `zpool/data/home` into `/home`, which again gets auto-prepended by `/mnt`. 
+1. We `zfs mount -a` which automounts `zpool/data/home` into `/home`, which again gets auto-prepended by `/mnt`.
 1. We lastly mount our EFI partition into `/mnt/efi`.
 1. We instruct ZFS to save its pool configuration via `zpool set cachefile=/etc/zfs/zpool.cache zpool`.
 
@@ -341,14 +601,146 @@ find '/opt/git.bak' -type d -empty -delete
 
 An example `zpool/data` dataset may now look like so:
 ```
-# zfs list -r -oname,mountpoint,canmount zpool/data
-NAME                MOUNTPOINT  CANMOUNT
-zpool/data          /           off
-zpool/data/home     /home       on
-zpool/data/opt      /opt        off
-zpool/data/opt/git  /opt/git    on
+# zfs list -r -oname,mountpoint,canmount,mounted zpool/data
+NAME                MOUNTPOINT       CANMOUNT  MOUNTED
+zpool/data          /                off       no
+zpool/data/home     /home            on        yes
+zpool/data/opt      /opt             off       no
+zpool/data/opt/git  /opt/git         on        yes
 ```
 
+## Nested environment-independent datasets
+
+### Caution
+
+If you want a dedicated dataset for a directory that lives deeper in your file system tree than just `/opt/git`, for example like `/var/lib/docker` make sure to not recursively create this structure in a single `zfs create` command.
+
+In [Adding another boot environment-independent dataset](#adding-another-boot-environment-independent-dataset) above you can safely do:
+```
+zfs create -o canmount=off zpool/data/opt
+```
+Here `zpool/data` already exists, you're only creating one child dataset `opt` and you're setting `-o canmount=off` so that it never mounts into your `/opt` directory.
+
+Now consider the same setup for `/var/lib/docker`. If you follow the exact same approach:
+```
+zfs create -o canmount=off zpool/data/var/lib
+```
+Docker will correctly report:
+```
+cannot create 'zpool/data/var/lib': parent does not exist
+```
+You might want to just create the parent then with `-p` argument:
+```
+zfs create -p -o canmount=off zpool/data/var/lib
+           ~~
+```
+Note, however, that `-o canmount=off` only applies to `lib` dataset and that `zpool/data/var` has just been auto-mounted into `/var`:
+```
+# zfs list -r -oname,mountpoint,canmount,mounted zpool/data
+NAME                MOUNTPOINT  CANMOUNT  MOUNTED
+zpool/data          /           off       no
+zpool/data/home     /home       on        yes
+zpool/data/opt      /opt        off       no
+zpool/data/opt/git  /opt/git    on        yes
+zpool/data/var      /var        on        yes  <---
+zpool/data/var/lib  /var/lib    off       no
+```
+
+### Advice
+
+Instead create nested parents in multiple steps where you set each one to `-o canmount=off`:
+```
+zfs create -o canmount=off zpool/data/var
+zfs create -o canmount=off zpool/data/var/lib
+```
+Lastly create the dataset you want mounted:
+```
+zfs create zpool/data/var/lib/docker
+```
+
+## Mounting zpool for maintenance
+
+In case you want to mount your zpool on an external operating system such as an Arch Linux live CD ISO image do it like so:
+
+```
+zpool import zpool -d /dev/disk/by-partuuid -R /mnt -f -N
+zfs load-key -L prompt zpool
+zfs mount zpool/root/archlinux
+zfs mount -a
+
+# UEFI system ...
+mount /dev/sda1 /mnt/efi
+
+# ... or legacy BIOS system
+mount /dev/sda1 /mnt/boot/syslinux
+
+arch-chroot /mnt /bin/bash
+```
+
+When done exit `chroot` and cleanly remove your work:
+
+```
+# UEFI system ...
+umount /mnt/efi
+
+# ... or legacy BIOS system
+umount /mnt/boot/syslinux
+
+zfs umount -a
+zpool export zpool
+```
+
+Explanation:
+
+- We always want to mount pools `by-partuuid` for consistency so we specifically only look for pools at `/dev/disk/by-partuuid`.
+- We mount our zpool with `-R /mnt` (aka `-o cachefile=none -o altroot=/mnt`). The pool is never cached, i.e. it's considered temporary. All pool and dataset mount paths have `/mnt` prepended. From `man zpoolprops`:
+    > This can be used when examining an unknown pool where the mount points cannot be trusted, or in an alternate boot environment, where the typical paths are not valid. `altroot` is not a persistent property. It is valid only while the system is up.
+- With `-f` and `-N` we force-mount our pool (`-f`) even if it previously wasn't cleanly exported; and we do not auto-mount any of its datasets (`-N`), not even the ones that have `canmount=on` set.
+
+    ```
+    # zfs list -oname,mountpoint,canmount,mounted
+    NAME                  MOUNTPOINT  CANMOUNT  MOUNTED
+    zpool                 none        off       no
+    zpool/data            /mnt        off       no
+    zpool/data/home       /mnt/home   on        no <-- Not immediately mounted
+    zpool/root            none        off       no
+    zpool/root/archlinux  /mnt        noauto    no <-- Not immediately mounted
+    ```
+- We load the decryption key by temporarily overriding the `keylocation` property to `-L prompt`. The default value is `file:///etc/zfs/zpool.key` which in all likelihood doesn't exist in this environment.
+- We mount our desired boot environment with `zfs mount zpool/root/archlinux`
+
+    ```
+    # zfs list -oname,mountpoint,canmount,mounted
+    NAME                  MOUNTPOINT  CANMOUNT  MOUNTED
+    zpool                 none        off       no
+    zpool/data            /mnt        off       no
+    zpool/data/home       /mnt/home   on        no
+    zpool/root            none        off       no
+    zpool/root/archlinux  /mnt        noauto    yes <-- Only boot env now mounted
+    ```
+- We mount all child datasets with `zfs mount -a` making `/mnt/home` available as well as any others you may have created yourself.
+
+    ```
+    # zfs list -oname,mountpoint,canmount,mounted
+    NAME                  MOUNTPOINT  CANMOUNT  MOUNTED
+    zpool                 none        off       no
+    zpool/data            /mnt        off       no
+    zpool/data/home       /mnt/home   on        yes <-- Now mounted
+    zpool/root            none        off       no
+    zpool/root/archlinux  /mnt        noauto    yes <-- Now mounted
+    ```
+- We lastly mount our EFI System Partition (ESP), in this example it's living at `/dev/sda1` so adjust this path accordingly.
+
+    ```
+    # df -hTP
+    Filesystem           Type      Size  Used Avail Use% Mounted on
+    ...                  ...       ...    ...   ...  ... ...
+    zpool/root/archlinux zfs       8.6G  2.5G  6.2G  29% /mnt
+    zpool/data/home      zfs       6.3G  161M  6.2G   3% /mnt/home
+    /dev/sda1            vfat      511M   31M  481M   6% /mnt/efi
+    ```
+- We're ready to `arch-chroot` into our boot environment.
+
 # Development
 
 ## Conventional commits
@@ -368,10 +760,10 @@ Commit _types_ besides `fix` and `feat` are:
 
 The following _scopes_ are known for this project. A Conventional Commits commit message may optionally use one of the following scopes or none:
 
-- `iso`: Changing Arch Linux ISO CD
+- `iso`: Changing Arch Linux live CD ISO image
 - `zbm`: Adjusting ZFSBootMenu's behavior
 - `zfs`: A change to how ZFS interacts with the system, either a pool or a dataset
-- `os`: Getting an perating system set up to correctly work in a ZFS boot environment
+- `os`: Getting an operating system set up to correctly work in a ZFS boot environment
 - `meta`: Affects the project's repo layout, readme content, file names etc.
 
 # Credits