GUIDE · PREVIEW
EDIT ON GITHUB →
GUIDE / SER.59
source: docs/guide/services/Reconciler.md
Services

Reconciler

Role

The reconciler drives workload lifecycle: starting, stopping, and migrating containers and VMs based on the org's desired state. It implements the level-triggered reconciliation pattern (the thermostat model from 09 Running Workloads) -- read desired state, read actual state, fix the difference.

The reconciler runs as a tier 1 node service, separate from the maintainer.

What It Does

  • Level-triggered loop: Periodically reads WorkloadDesired from the maintainer (via IPC on localhost:7208), compares to what's actually running, and converges.
  • Container management (tier 2): Starts org containers via unshare + pivot_root + cgroup v2. Sets up encrypted scratch volumes. Manages the container lifecycle (start, stop, restart on crash).
  • VM management (tier 3-4): Starts org VMs and user VMs via cloud-hypervisor REST API. Assembles VM storage (base image + qcow2 COW overlay). Creates TAP interfaces for network profiles. Manages live migration (drain before upgrade), snapshot/restore (degraded path), and incremental state sync (qcow2 overlay diffing or dm-snapshot dirty block tracking to stream changes to org shard storage).
  • Observed state reporting: Reports what's actually running back to the maintainer via IPC. The maintainer writes it to the WorkloadObserved CRDT, which gossips to all nodes.
  • Scratch encryption: Sets up per-service dm-crypt encrypted scratch volumes. Keys come from the Key Service via the key derivation chain.
  • Placement constraints: Evaluates workload manifests' placement rules (spread, affinity, anti-affinity) against the Topology Map when deciding whether this node should run a workload.

Why It's Separate from the Maintainer

The maintainer must be stable -- it's the gossip participant, the CRDT owner, the health signal. If the maintainer crashes, the node looks dead to the org.

The reconciler does risky work: starting untrusted containers, managing VM processes, handling driver issues. A reconciler crash affects workloads but not the node's org membership. s6 restarts the reconciler; the maintainer stays running.

Separation also enforces the silo'd design: the reconciler has no CRDT code, no gossip code, no WireGuard code. It's a pure client of the maintainer's IPC interface.

Links