================== Provisioning disks ================== As a tech preview, Warewulf provides structures to define disks, partitions, and file systems. These structures can generate a configuration for `Ignition`_ to provision partitions and file systems dynamically on cluster nodes, or with sfdisk, mkfs, and mkswap during boot. .. _Ignition: https://coreos.github.io/ignition/ Ignition can, for example, create ``swap`` partitions or ``/scratch`` file systems. Requirements ============ Partition and file system creation requires both ``ignition`` and ``sgdisk`` to be installed in the image. Rocky Linux ----------- .. code-block:: shell dnf install ignition gdisk .. note:: Packages for Ignition are not currently available for Rocky Linux 8, but it is available for Rocky Linux 9 as part of "appstream." openSuse Leap ------------- .. code-block:: shell zypper install ignition gptfdisk Disks and partitions ==================== A node or profile can have several disks. Each disk is identified by the path to its block device. Each disk holds a map to its partitions and a ``bool`` switch to indicate if an existing, non-matching partition table should be overwritten. Each partition is identified by its label. The partition number can be omitted, but specifying it is recommended as Ignition may fail without it. Partition sizes should also be set (specified in MiB), except for the last partition: if no size is given, the maximum available size is used. Each partition has the switches ``should_exist`` and ``wipe_partition_entry`` which control the partition creation process (via the ``--partcreate`` and ``--partwipe`` flags). When omitting a partition number the ``wipe_partition_entry`` should be true, as this allows ignition to replace the existing partition. .. code-block:: shell wwctl node set n1 \ --diskname /dev/vda --diskwipe \ --partname scratch --partcreate --partwipe --partnumber 1 File systems ============ File systems are identified by their underlying block device, preferably using the ``/dev/by-partlabel`` format. Except for a ``swap`` partition, an absolute path for the mount point must be specified for each file system. Depending on the image used, valid formats are ``btrfs``, ``ext3``, ``ext4``, and ``xfs``. Each file system has the switch ``wipe_filesystem`` to control whether an existing file system is wiped. .. code-block:: shell wwctl node set n1 \ --diskname /dev/vda --partname scratch \ --fsname scratch --fsformat btrfs --fspath /scratch Boot-time configuration ======================= Ignition uses systemd, as the underlying ``sgdisk`` command relies on dbus notifications. 1. ``ignition-disks-ww4.service`` uses Ignition to create the specified partitions and file systems. 2. ``ww4-disks.target`` depends on a matching ``.mount`` unit for each mounted file system. 3. Each ``.mount`` creates the necessary mount points in the root file system and mounts the provisioned file systems during boot. These services and mount units are generated by the ``ignition`` overlay and depend on the existence of the file ``/warewulf/ignition.json``, also generated by the ``ignition`` overlay. Example disk configurations =========================== This command formats a btrfs file system on a "scratch" partition of "vda" and mounts it at ``/scratch``. .. code-block:: shell wwctl node set n1 \ --diskname /dev/vda --diskwipe \ --partname scratch --partcreate --partnumber 1 \ --fsname scratch --fsformat btrfs --fspath /scratch This command adds a swap partition to the "vda" disk. .. code-block:: shell wwctl node set n1 \ --diskname /dev/vda \ --partname swap --partsize=1024 --partnumber 2 \ --fsname swap --fsformat swap --fspath swap Re-using or wiping disks ======================== For empty disks the desired configuration is created and the filesystems are mounted. If partitions or file systems already exist on the disk, ``ignition`` tries to reuse existing file systems by default. To ignore existing file systems and provision fresh file systems on each boot, specify the ``--fswipe`` flag for that filesystem, and ``--diskwipe`` for the disk, as necessary. If you would like to re-use existing partitions but want to replace existing file systems once, you may * wipe the existing data with tools like ``wipefs`` or ``dd`` [#]_; or * set the ``--fswipe`` flag and remove it after one reboot. .. [#] With ``wipefs`` you have to remove the filesystem *and* partition information. E.g., use ``wipefs -fa /dev/vda*`` to remove all file system information and partition information. See the `upstream ignition documentation`_ for additional information. .. _upstream ignition documentation: https://coreos.github.io/ignition/operator-notes/#filesystem-reuse-semantics .. _swap-and-image-memory: Swap and image memory usage =========================== Warewulf images run entirely in memory. Configuring a local swap partition can allow the kernel to reclaim that RAM for applications — but only under the right conditions. Whether swap can free image memory depends on which root filesystem type the node uses. tmpfs root ---------- When the root filesystem is ``tmpfs`` (the default for two-stage dracut boot, or when ``--root=tmpfs`` is set explicitly), the image lives in the page cache. The Linux kernel can swap ``tmpfs`` pages to a local swap device exactly as it would any other anonymous memory. Adding swap therefore lets the kernel evict cold image pages to disk and reclaim that RAM for running workloads. This is the recommended configuration for nodes with large images relative to available RAM. initramfs root (single-stage boot default) ------------------------------------------ The default single-stage boot places the image in an ``initramfs`` root, which is an instance of ``ramfs``. Unlike ``tmpfs``, ``ramfs`` pages are pinned in memory: the kernel will never swap them out. Configuring swap on a node with an ``initramfs`` root will **not** free any memory used by the image. If you are using single-stage boot and want swap to help with image memory, switch to ``tmpfs`` root first: .. code-block:: shell wwctl profile set default --root=tmpfs For background on ``tmpfs`` NUMA interleaving and size limits, see :ref:`tmpfs-and-numa`. .. note:: Two-stage dracut boot always uses ``tmpfs`` regardless of the ``--root`` setting. If you are already using dracut, no additional root configuration is needed. Example: configuring swap to reclaim image memory -------------------------------------------------- This example provisions a swap partition on the local disk of each node and activates it at boot, enabling the kernel to reclaim RAM occupied by the node image. **1. Configure tmpfs root** (skip if using two-stage dracut boot): .. code-block:: shell wwctl profile set default --root=tmpfs **2. Add a swap partition** to the node or profile disk configuration. This example adds an 8 GiB swap partition as the first partition of ``/dev/vda``: .. code-block:: shell wwctl profile set default \ --diskname /dev/vda --diskwipe \ --partname swap --partcreate --partnumber 1 --partsize=8192 \ --fsname swap --fsformat swap --fspath swap **3. Add the required overlays** to the system overlay so that the swap partition is formatted and activated at boot. The ``-O`` / ``--system-overlays`` flag replaces the entire overlay list, so include any overlays already configured for the profile. Use Ignition (recommended for Rocky Linux 9 and openSUSE): .. code-block:: shell wwctl profile set default \ -O wwinit,wwclient,fstab,hostname,ssh.host_keys,systemd.netname,NetworkManager,ignition,systemd.swap Or, for systems without Ignition (e.g., Rocky Linux 8), use the ``sfdisk`` and ``mkswap`` overlays instead: .. code-block:: shell wwctl profile set default \ -O wwinit,wwclient,fstab,hostname,ssh.host_keys,systemd.netname,NetworkManager,sfdisk,mkswap,systemd.swap **4. Rebuild the dracut initramfs** with the tools needed to provision the disk during the two-stage boot (skip for single-stage boot): .. code-block:: shell # Ignition path wwctl image exec rockylinux-9 -- /usr/bin/dracut --force --no-hostonly \ --add wwinit --add ignition --regenerate-all # sfdisk/mkswap path wwctl image exec rockylinux-8 -- /usr/bin/dracut --force --no-hostonly \ --add wwinit --install sfdisk --install blockdev --install udevadm \ --install mkswap --regenerate-all **5. Reboot nodes** to apply the new configuration. Verifying swap is active ------------------------ After reboot, confirm that the swap device is active: .. code-block:: shell swapon --show Then check the overall memory picture with ``free -h``: .. code-block:: text $ free -h total used free shared buff/cache available Mem: 15Gi 12Gi 256Mi 120Mi 2.8Gi 2.8Gi Swap: 8.0Gi 0Ki 8.0Gi At idle, most of the node's RAM is occupied by the image. The ``Swap:`` line shows the available device but near-zero usage at this point. Confirming that the image gets swapped out ------------------------------------------ To demonstrate that the kernel evicts image pages to swap when an application needs memory, first note the image size with ``df -h /``: .. code-block:: shell df -h / This shows how much tmpfs space the image occupies — that is the amount of RAM currently holding the image. Now apply memory pressure using ``stress-ng`` (install it in the OS image if not already present). The allocation must exceed **available** RAM — not just total RAM — to force the kernel to evict image pages. Compute the target from ``MemTotal``: .. code-block:: shell stress-ng --vm 1 \ --vm-bytes $(awk '/MemTotal/{printf "%dM", $2*0.9/1024}' /proc/meminfo) \ --vm-keep -t 60s & While ``stress-ng`` is running, observe memory usage: .. code-block:: shell free -h .. code-block:: text $ free -h total used free shared buff/cache available Mem: 15Gi 15Gi 32Mi 120Mi 512Mi 192Mi Swap: 8.0Gi 4.2Gi 3.8Gi The ``Swap: used`` value has grown by roughly the size of the image. The kernel has evicted cold image pages to swap, making that RAM available to the application. The application can access the full physical memory of the node, not just what is left over after the image is loaded. After ``stress-ng`` finishes, the evicted image pages remain on swap until they are accessed again, so ``free -h`` will continue to show swap usage until the node is under less pressure and pages are faulted back in as needed. Moving image pages to swap proactively -------------------------------------- Rather than simulating a workload, you can instruct the kernel to push image pages to swap directly. On Linux 6.1 and later with cgroup v2, write the desired reclaim size to ``memory.reclaim``: .. code-block:: shell # Request that the kernel reclaim up to 4 GiB from the root cgroup # (adjust to match your image size shown by df -h /) echo "4G" > /sys/fs/cgroup/memory.reclaim If the kernel cannot reclaim the full requested amount it returns an error: .. code-block:: text -bash: echo: write error: Resource temporarily unavailable This is expected and benign. It means the kernel reclaimed as much as it could but the reclaimable pool was exhausted before reaching the target. The kernel will swap out reclaimable pages until the target is met or exhausted, then stop. Run ``free -h`` immediately afterwards to see the reduction in ``Mem: used`` and corresponding increase in ``Swap: used``. This works best when run early in the boot process, before applications start, when nearly all anonymous memory belongs to the image. To automate it, add a systemd unit to a custom overlay that runs at ``local-fs.target``: .. code-block:: ini [Unit] Description=Reclaim OS image memory to swap After=local-fs.target ConditionPathExists=/sys/fs/cgroup/memory.reclaim [Service] Type=oneshot ExecStart=/bin/sh -c 'echo "$(df --output=used / | tail -1)K" > /sys/fs/cgroup/memory.reclaim' [Install] WantedBy=multi-user.target This reads the actual image size from ``df`` and requests that exact amount be reclaimed, so it adapts automatically as the image grows or shrinks. .. _provision to disk: Provision to disk ================= *New in Warewulf v4.6.2* As a tech preview, the Warewulf two-stage boot process can provision the node image to local storage. .. warning:: This functionality is a technology preview and should be used with care. Pay specific attention to ``wipeFilesytem`` and similar settings. .. note:: Warewulf doesn't install a bootloader to the disk or add UEFI entries. Nodes still request an image and configuration from the Warewulf server on every boot. .. note:: While provisioning to disk should be possible during a single-stage boot, not all features are available: - Warewulf does not perform hardware detection to ensure that necessary kernel modules are loaded prior to init. - Warewulf does not load udev to ensure that ``/dev/disk/by-*`` symlinks are available prior to init. With Ignition ------------- Warewulf needs a prepared file system to deploy the image to. Warewulf can provision this file system using Ignition. To use Ignition, include ``ignition`` in your system overlay. The ignition overlay provisions disks during init and, optionally, during the first stage of a two-stage boot. This allows the root file system to be provisioned before the image is loaded. .. code-block:: shell wwctl node set wwnode1 \ --diskname /dev/vda --diskwipe \ --partname rootfs --partcreate --partnumber 1 \ --fsname rootfs --fsformat ext4 --fspath / In order to allow Dracut to provision the disk, partition, and file system, Ignition must be included in the Dracut image. .. code-block:: shell wwctl image exec rockylinux-9 -- /usr/bin/dracut --force --no-hostonly --add wwinit --add ignition --regenerate-all The necessary file system may alternatively be prepared out-of-band. With sfdisk and mkfs -------------------- Systems that do not have access to Ignition (e.g., Rocky Linux 8) can provision the root file system using a combination of ``sfdisk`` and ``mkfs``. To use them, include ``sfdisk`` and ``mkfs`` in your system overlay. The ``sfdisk`` and ``mkfs`` overlays provision disk and file systems during the first stage of a two-stage boot. This allows the root file system to be provisioned before the image is loaded. Configure the ``sfdisk`` and ``mkfs`` overlays using resources: .. code-block:: shell wwctl node set wwnode1 \ --diskname /dev/vda --diskwipe \ --partname rootfs --partcreate --partnumber 1 \ --fsname rootfs --fsformat ext4 --fspath / In order to allow Dracut to provision the disk, partition, and file system, some additional commands must be included in the Dracut image, depending on which functionality is used: - **sfdisk:** writes the partition table - **blockdev:** used to re-read the partition table after writing - **udevadm:** used to trigger udev events after writing the partition table - **mkfs:** formats file systems (may also require file-system-specific commands like mkfs.ext4) - **mkfs.ext4**, **mkfs.btrfs**, etc: used by mkfs to format specific file systems - **wipefs:** used to determine if a file system already exists .. code-block:: shell wwctl image exec rockylinux-8 -- /usr/bin/dracut --force --no-hostonly \ --add wwinit \ --install sfdisk \ --install blockdev \ --install udevadm \ --install mkfs \ --install mkfs.ext4 \ --install wipefs \ --regenerate-all Configuring the root device --------------------------- Set the desired storage device for the image using the ``--root`` parameter. .. code-block:: shell wwctl node set wwnode1 --root /dev/disk/by-partlabel/rootfs Known Problems -------------- If the partition table on the disk isn't properly readable the command ``sgdisk --zap-all`` (which is used by Ignition to wipe the partition table) returns with code ``2``. This is interpreted by Ignition < ``2.16.2`` as an error, and no partitions or filesystems are created. Since the partition table is still wiped, partitioning and formatting should succeed on the next boot.