From 1fd8b65ebf34112cd4b5d48be2d3d789b02271bb Mon Sep 17 00:00:00 2001 From: Jonathon Anderson Date: Tue, 17 Mar 2026 17:43:01 -0600 Subject: [PATCH] Document reclaiming RAM by moving the image to swap Signed-off-by: Jonathon Anderson --- CHANGELOG.md | 1 + userdocs/nodes/disks.rst | 218 +++++++++++++++++++ userdocs/overlays/overlays.rst | 4 + userdocs/troubleshooting/troubleshooting.rst | 5 + 4 files changed, 228 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e561ba9..ca451188 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added +- Document reclaiming RAM by moving the image to swap - New `--partwipe` flag for profile and node set - Updated arguments for `ValidString` to match `regexp.MatchString` - New `mig` overlay to configure NVIDIA MIG devices. #2102 diff --git a/userdocs/nodes/disks.rst b/userdocs/nodes/disks.rst index 5e5a84f9..c9fb6f3c 100644 --- a/userdocs/nodes/disks.rst +++ b/userdocs/nodes/disks.rst @@ -148,6 +148,224 @@ See the `upstream ignition documentation`_ for additional information. .. _upstream ignition documentation: https://coreos.github.io/ignition/operator-notes/#filesystem-reuse-semantics +.. _swap-and-image-memory: + +Swap and image memory usage +=========================== + +Warewulf node images run entirely in memory. Configuring a local swap partition +can allow the kernel to reclaim that RAM for applications — but only under the +right conditions. Whether swap can free image memory depends on which root +filesystem type the node uses. + +tmpfs root +---------- + +When the root filesystem is ``tmpfs`` (the default for two-stage dracut boot, or +when ``--root=tmpfs`` is set explicitly), the node image lives in the page cache. +The Linux kernel can swap ``tmpfs`` pages to a local swap device exactly as it +would any other anonymous memory. Adding swap therefore lets the kernel evict +cold image pages to disk and reclaim that RAM for running workloads. + +This is the recommended configuration for nodes with large images relative to +available RAM. + +initramfs root (single-stage boot default) +------------------------------------------ + +The default single-stage boot places the image in an ``initramfs`` root, which +is an instance of ``ramfs``. Unlike ``tmpfs``, ``ramfs`` pages are pinned in +memory: the kernel will never swap them out. Configuring swap on a node with an +``initramfs`` root will **not** free any memory used by the node image. + +If you are using single-stage boot and want swap to help with image memory, +switch to ``tmpfs`` root first: + +.. code-block:: shell + + wwctl profile set default --root=tmpfs + +For background on ``tmpfs`` NUMA interleaving and size limits, see +:ref:`tmpfs-and-numa`. + +.. note:: + + Two-stage dracut boot always uses ``tmpfs`` regardless of the ``--root`` + setting. If you are already using dracut, no additional root configuration + is needed. + +Example: configuring swap to reclaim image memory +-------------------------------------------------- + +This example provisions a swap partition on the local disk of each node and +activates it at boot, enabling the kernel to reclaim RAM occupied by the node +image. + +**1. Configure tmpfs root** (skip if using two-stage dracut boot): + +.. code-block:: shell + + wwctl profile set default --root=tmpfs + +**2. Add a swap partition** to the node or profile disk configuration. This +example adds an 8 GiB swap partition as the first partition of ``/dev/vda``: + +.. code-block:: shell + + wwctl profile set default \ + --diskname /dev/vda --diskwipe \ + --partname swap --partcreate --partnumber 1 --partsize=8192 \ + --fsname swap --fsformat swap --fspath swap + +**3. Add the required overlays** to the system overlay so that the swap +partition is formatted and activated at boot. The ``-O`` / ``--system-overlays`` +flag replaces the entire overlay list, so include any overlays already +configured for the profile. Use Ignition (recommended for Rocky Linux 9 and +openSUSE): + +.. code-block:: shell + + wwctl profile set default \ + -O wwinit,wwclient,fstab,hostname,ssh.host_keys,systemd.netname,NetworkManager,ignition,systemd.swap + +Or, for systems without Ignition (e.g., Rocky Linux 8), use the ``sfdisk`` and +``mkswap`` overlays instead: + +.. code-block:: shell + + wwctl profile set default \ + -O wwinit,wwclient,fstab,hostname,ssh.host_keys,systemd.netname,NetworkManager,sfdisk,mkswap,systemd.swap + +**4. Rebuild the dracut initramfs** with the tools needed to provision the disk +during the two-stage boot (skip for single-stage boot): + +.. code-block:: shell + + # Ignition path + wwctl image exec rockylinux-9 -- /usr/bin/dracut --force --no-hostonly \ + --add wwinit --add ignition --regenerate-all + + # sfdisk/mkswap path + wwctl image exec rockylinux-8 -- /usr/bin/dracut --force --no-hostonly \ + --add wwinit --install sfdisk --install blockdev --install udevadm \ + --install mkswap --regenerate-all + +**5. Reboot nodes** to apply the new configuration. + +Verifying swap is active +------------------------ + +After reboot, confirm that the swap device is active: + +.. code-block:: shell + + swapon --show + +Then check the overall memory picture with ``free -h``: + +.. code-block:: text + + $ free -h + total used free shared buff/cache available + Mem: 15Gi 12Gi 256Mi 120Mi 2.8Gi 2.8Gi + Swap: 8.0Gi 0Ki 8.0Gi + +At idle, most of the node's RAM is occupied by the image. The ``Swap:`` line +shows the available device but near-zero usage at this point. + +Confirming that the image gets swapped out +------------------------------------------ + +To demonstrate that the kernel evicts image pages to swap when an application +needs memory, first note the image size with ``df -h /``: + +.. code-block:: shell + + df -h / + +This shows how much tmpfs space the image occupies — that is the amount of RAM +currently holding the node image. + +Now apply memory pressure using ``stress-ng`` (install it in the node image if +not already present). The allocation must exceed **available** RAM — not just +total RAM — to force the kernel to evict image pages. Compute the target from +``MemTotal``: + +.. code-block:: shell + + stress-ng --vm 1 \ + --vm-bytes $(awk '/MemTotal/{printf "%dM", $2*0.9/1024}' /proc/meminfo) \ + --vm-keep -t 60s & + +While ``stress-ng`` is running, observe memory usage: + +.. code-block:: shell + + free -h + +.. code-block:: text + + $ free -h + total used free shared buff/cache available + Mem: 15Gi 15Gi 32Mi 120Mi 512Mi 192Mi + Swap: 8.0Gi 4.2Gi 3.8Gi + +The ``Swap: used`` value has grown by roughly the size of the node image. The +kernel has evicted cold image pages to swap, making that RAM available to the +application. The application can access the full physical memory of the node, +not just what is left over after the image is loaded. + +After ``stress-ng`` finishes, the evicted image pages remain on swap until +they are accessed again, so ``free -h`` will continue to show swap usage until +the node is under less pressure and pages are faulted back in as needed. + +Moving image pages to swap proactively +--------------------------------------- + +Rather than simulating a workload, you can instruct the kernel to push image +pages to swap directly. On Linux 6.1 and later with cgroup v2, write the +desired reclaim size to ``memory.reclaim``: + +.. code-block:: shell + + # Request that the kernel reclaim up to 4 GiB from the root cgroup + # (adjust to match your image size shown by df -h /) + echo "4G" > /sys/fs/cgroup/memory.reclaim + +If the kernel cannot reclaim the full requested amount it returns an error: + +.. code-block:: text + + -bash: echo: write error: Resource temporarily unavailable + +This is expected and benign. It means the kernel reclaimed as much as it could +but the reclaimable pool was exhausted before reaching the target. + +The kernel will swap out reclaimable pages until the target is met or exhausted, then stop. +Run ``free -h`` immediately afterwards to see the reduction in ``Mem: used`` +and corresponding increase in ``Swap: used``. + +This works best when run early in the boot process, before applications start, +when nearly all anonymous memory belongs to the image. To automate it, add a +systemd unit to a custom overlay that runs at ``local-fs.target``: + +.. code-block:: ini + + [Unit] + Description=Reclaim node image memory to swap + After=local-fs.target + ConditionPathExists=/sys/fs/cgroup/memory.reclaim + + [Service] + Type=oneshot + ExecStart=/bin/sh -c 'echo "$(df --output=used / | tail -1)K" > /sys/fs/cgroup/memory.reclaim' + + [Install] + WantedBy=multi-user.target + +This reads the actual image size from ``df`` and requests that exact amount be +reclaimed, so it adapts automatically as the image grows or shrinks. + .. _provision to disk: Provision to disk diff --git a/userdocs/overlays/overlays.rst b/userdocs/overlays/overlays.rst index a95c0d1c..0ee6a0ac 100644 --- a/userdocs/overlays/overlays.rst +++ b/userdocs/overlays/overlays.rst @@ -520,6 +520,10 @@ include wipefs to detect existing file systems. wwctl image exec rockylinux-9 -- /usr/bin/dracut --force --no-hostonly --add wwinit --install mkswap --regenerate-all +For a complete walkthrough of configuring swap to free memory consumed by the +in-memory node image, including which root filesystem types support swap +reclamation, see :ref:`swap-and-image-memory`. + systemd mounts -------------- diff --git a/userdocs/troubleshooting/troubleshooting.rst b/userdocs/troubleshooting/troubleshooting.rst index 9261046f..ad584e3c 100644 --- a/userdocs/troubleshooting/troubleshooting.rst +++ b/userdocs/troubleshooting/troubleshooting.rst @@ -251,6 +251,11 @@ argument: By default this is set to 50% of physical RAM. Note that tmpfs is required for SELinux overlays since initramfs cannot preserve SELinux contexts. +Because the root is ``tmpfs``, the kernel can also swap cold image pages to a +local swap device, freeing RAM for running workloads. This does not apply to the +default ``initramfs`` root (single-stage boot), where pages are pinned in memory +and cannot be swapped. See :ref:`swap-and-image-memory` for a complete walkthrough. + .. note:: On some systems, it may also be necessary to include the ``noefi`` kernel