GUIDE · PREVIEW
EDIT ON GITHUB →
GUIDE / CON.22
source: docs/guide/concepts/kexec.md
Concepts

kexec

What It Is

kexec (kernel execute) is a Linux system call and mechanism that loads a new kernel into memory and transfers control to it, completely bypassing the firmware and bootloader. It's a "warm reboot" -- the hardware stays initialized, no POST, no UEFI, no bootloader. Just: old kernel -> new kernel.

Why It Matters

A normal reboot goes through the entire firmware initialization sequence: POST (seconds to minutes on servers), UEFI DXE phase (loading all drivers), bootloader, then the new kernel. On server hardware, this can take 30-60 seconds or more.

kexec skips all of that. The transition takes seconds. For FortrOS, this means:

  • The preboot can load an optimized kernel almost instantly
  • Rolling upgrades switch kernels with minimal downtime
  • Per-host kernel optimization is practical (each host boots a kernel tuned for its hardware without paying the reboot cost every time)

How It Works

The Two-Step Process

kexec operates in two phases:

Load: Parse and prepare the new kernel + initramfs in memory.

kexec -l /path/to/vmlinuz \
  --initrd=/path/to/initramfs \
  --command-line="root=/dev/vda2 quiet"

This stage validates the kernel image format, allocates memory for the new kernel, and prepares the transition but does not execute it. The old kernel is still running.

Execute: Shut down the old kernel and jump to the new one.

kexec -e

The old kernel tears down its state (stops interrupts, quiesces devices) and jumps to the new kernel's entry point. From the new kernel's perspective, it's a fresh boot -- it initializes its own subsystems, mounts filesystems, and starts its init process.

What Persists Across kexec

  • RAM contents: Physical memory is not cleared. The new kernel gets fresh virtual memory mappings but the physical pages from the old kernel are still there (until overwritten). This is how FortrOS passes the LUKS key -- it's in an initramfs concatenated with the new kernel's initramfs.
  • Hardware state: Devices remain in whatever state the old kernel left them. Well-behaved drivers quiesce their devices during the kexec shutdown sequence, but not all hardware handles this cleanly.
  • UEFI Runtime Services: Still available (they were set up by firmware during the original boot and persist in memory).

What Does NOT Persist

  • Filesystem mounts: The new kernel starts with no mounted filesystems (it boots into its initramfs, same as a normal boot).
  • Running processes: All processes from the old kernel are gone.
  • Network connections: All sockets and network state are gone.
  • Kernel modules: The new kernel loads its own modules from its initramfs.

Concatenated Initramfs

Linux supports concatenating multiple cpio archives as the initramfs. The kernel unpacks them in order -- later archives can add files or override earlier ones:

# Combine the generation's initramfs with a key-carrying archive
cat generation-initramfs.cpio.gz luks-key.cpio.gz > combined.cpio.gz
kexec -l vmlinuz --initrd=combined.cpio.gz

FortrOS uses this to append a minimal archive containing /luks.key to the generation's initramfs. The main OS's persist-mount service reads the key, unlocks /persist, and deletes the key file. The key exists in the initramfs (tmpfs) only -- it's never written to a persistent disk.

Security Considerations

No firmware verification on kexec: UEFI Secure Boot verifies the first kernel loaded by firmware. kexec bypasses firmware, so Secure Boot doesn't verify the kexec'd kernel. The old kernel must verify the new kernel itself -- FortrOS does this with Ed25519 signature verification before kexec.

Memory contents are accessible: Since RAM isn't cleared across kexec, a malicious new kernel could scan memory for secrets left by the old kernel. FortrOS mitigates this by: (1) only kexec'ing org-signed images, and (2) the key-carrying initramfs is the only secret, consumed and zeroed immediately.

Hardware quiesce issues: Some hardware (particularly GPUs and USB controllers) doesn't handle kexec gracefully. The old kernel's shutdown sequence may not fully quiesce all devices, leading to hangs or hardware errors in the new kernel. FortrOS's preboot environment is minimal (no GPU drivers, limited USB) which reduces this risk.

Kernel Lockdown and kexec

When the kernel lockdown module is active (common with Secure Boot), unsigned kexec is blocked (kexec_load requires CAP_SYS_BOOT and the kernel must allow it). FortrOS uses kexec_file_load (the newer interface) which supports signature verification by the kernel itself before loading.

How FortrOS Uses It

Preboot -> main OS only. kexec is used for exactly one transition: the preboot kernel loads and executes the generation kernel. The preboot verifies the generation's Ed25519 signature, builds the combined initramfs (generation + LUKS key), and kexec's. This is the core boot transition described in 05 Loading the Real OS.

The generation kernel has kexec disabled. CONFIG_KEXEC=n in the generation kernel config. This is a security hardening decision baked into the architecture, not a configuration toggle. If an attacker gains root on the main OS, they cannot kexec into a malicious kernel -- the syscall doesn't exist. Every generation change goes through a full reboot, which means the preboot re-executes: TPM authentication, TLS to the org, generation signature verification, LUKS key derivation. The entire chain of trust is re-established.

Rolling upgrades use full reboot. When a node upgrades to a new generation, it reboots. The preboot authenticates, selects the new generation (via the "send what you have" protocol from 04 Disk Encryption), and kexec's into it. VMs running on the node are live-migrated to other nodes before the reboot (the drain step from 10 Sustaining the Org). If no migration target is available and the upgrade is forced (current generation below the security floor), VMs are snapshot/restored -- a degraded path. Persistent VM state lives in shard storage; scratch is a local cache that can be rebuilt.

Recovery uses full reboot. If a generation fails (boot watchdog timeout), the node reboots through firmware and preboot. The preboot selects the previous healthy generation. Full reboot is correct here -- the failed kernel may have left hardware in a bad state that kexec wouldn't clean up.

Alternatives

Full reboot: Go through firmware and bootloader every time. Slower but guarantees clean hardware state and full trust chain re-establishment. FortrOS uses full reboot for all generation changes; kexec only for the preboot -> main OS transition within a single boot.

pivot_root: Instead of loading a new kernel, switch the root filesystem. Same kernel keeps running. Used by Talos for switching from init to the real OS. Simpler but doesn't allow per-host optimized kernels.

KHO (Kexec HandOver): A Linux kernel feature (6.16+, major refactor in 6.19) that preserves firmware context (ACPI tables, memory maps) across kexec. KHO is the right concept for FortrOS's preboot -> main OS transition: the generation kernel needs ACPI tables that only firmware provides, and KHO passes them cleanly. However, KHO is still maturing -- known crash bugs (CONFIG_DEFERRED_STRUCT_PAGE_INIT incompatibility) and active API changes mean it's not production-ready. FortrOS uses acpi_rsdp= (passing the ACPI root pointer address on the kernel command line) as the reliable fallback. When KHO stabilizes, it becomes the clean replacement.

LUO (Live Update Orchestrator): Built on KHO, preserves VM state across kexec for live host kernel updates. Incompatible with FortrOS's security model: LUO requires kexec on the main OS, which FortrOS disables (CONFIG_KEXEC=n). FortrOS handles VM continuity through live migration to other nodes (drain before reboot), not through in-place kernel replacement. The tradeoff: VMs experience a brief migration pause instead of zero-downtime kernel swap. This is acceptable because the security property (no kexec bypass of the preboot trust chain) is more valuable than the convenience.

Links