Document reclaiming RAM by moving the image to swap

Signed-off-by: Jonathon Anderson <janderson@ciq.com>

userdocs/nodes/disks.rst

@@ -148,6 +148,224 @@ See the `upstream ignition documentation`_ for additional information.
 .. _upstream ignition documentation: https://coreos.github.io/ignition/operator-notes/#filesystem-reuse-semantics
 
 
+.. _swap-and-image-memory:
+
+Swap and image memory usage
+===========================
+
+Warewulf node images run entirely in memory. Configuring a local swap partition
+can allow the kernel to reclaim that RAM for applications — but only under the
+right conditions. Whether swap can free image memory depends on which root
+filesystem type the node uses.
+
+tmpfs root
+----------
+
+When the root filesystem is ``tmpfs`` (the default for two-stage dracut boot, or
+when ``--root=tmpfs`` is set explicitly), the node image lives in the page cache.
+The Linux kernel can swap ``tmpfs`` pages to a local swap device exactly as it
+would any other anonymous memory. Adding swap therefore lets the kernel evict
+cold image pages to disk and reclaim that RAM for running workloads.
+
+This is the recommended configuration for nodes with large images relative to
+available RAM.
+
+initramfs root (single-stage boot default)
+------------------------------------------
+
+The default single-stage boot places the image in an ``initramfs`` root, which
+is an instance of ``ramfs``. Unlike ``tmpfs``, ``ramfs`` pages are pinned in
+memory: the kernel will never swap them out. Configuring swap on a node with an
+``initramfs`` root will **not** free any memory used by the node image.
+
+If you are using single-stage boot and want swap to help with image memory,
+switch to ``tmpfs`` root first:
+
+.. code-block:: shell
+
+   wwctl profile set default --root=tmpfs
+
+For background on ``tmpfs`` NUMA interleaving and size limits, see
+:ref:`tmpfs-and-numa`.
+
+.. note::
+
+   Two-stage dracut boot always uses ``tmpfs`` regardless of the ``--root``
+   setting. If you are already using dracut, no additional root configuration
+   is needed.
+
+Example: configuring swap to reclaim image memory
+--------------------------------------------------
+
+This example provisions a swap partition on the local disk of each node and
+activates it at boot, enabling the kernel to reclaim RAM occupied by the node
+image.
+
+**1. Configure tmpfs root** (skip if using two-stage dracut boot):
+
+.. code-block:: shell
+
+   wwctl profile set default --root=tmpfs
+
+**2. Add a swap partition** to the node or profile disk configuration. This
+example adds an 8 GiB swap partition as the first partition of ``/dev/vda``:
+
+.. code-block:: shell
+
+   wwctl profile set default \
+     --diskname /dev/vda --diskwipe \
+     --partname swap --partcreate --partnumber 1 --partsize=8192 \
+     --fsname swap --fsformat swap --fspath swap
+
+**3. Add the required overlays** to the system overlay so that the swap
+partition is formatted and activated at boot. The ``-O`` / ``--system-overlays``
+flag replaces the entire overlay list, so include any overlays already
+configured for the profile. Use Ignition (recommended for Rocky Linux 9 and
+openSUSE):
+
+.. code-block:: shell
+
+   wwctl profile set default \
+     -O wwinit,wwclient,fstab,hostname,ssh.host_keys,systemd.netname,NetworkManager,ignition,systemd.swap
+
+Or, for systems without Ignition (e.g., Rocky Linux 8), use the ``sfdisk`` and
+``mkswap`` overlays instead:
+
+.. code-block:: shell
+
+   wwctl profile set default \
+     -O wwinit,wwclient,fstab,hostname,ssh.host_keys,systemd.netname,NetworkManager,sfdisk,mkswap,systemd.swap
+
+**4. Rebuild the dracut initramfs** with the tools needed to provision the disk
+during the two-stage boot (skip for single-stage boot):
+
+.. code-block:: shell
+
+   # Ignition path
+   wwctl image exec rockylinux-9 -- /usr/bin/dracut --force --no-hostonly \
+     --add wwinit --add ignition --regenerate-all
+
+   # sfdisk/mkswap path
+   wwctl image exec rockylinux-8 -- /usr/bin/dracut --force --no-hostonly \
+     --add wwinit --install sfdisk --install blockdev --install udevadm \
+     --install mkswap --regenerate-all
+
+**5. Reboot nodes** to apply the new configuration.
+
+Verifying swap is active
+------------------------
+
+After reboot, confirm that the swap device is active:
+
+.. code-block:: shell
+
+   swapon --show
+
+Then check the overall memory picture with ``free -h``:
+
+.. code-block:: text
+
+   $ free -h
+                  total        used        free      shared  buff/cache   available
+   Mem:            15Gi        12Gi       256Mi       120Mi       2.8Gi       2.8Gi
+   Swap:           8.0Gi        0Ki       8.0Gi
+
+At idle, most of the node's RAM is occupied by the image. The ``Swap:`` line
+shows the available device but near-zero usage at this point.
+
+Confirming that the image gets swapped out
+------------------------------------------
+
+To demonstrate that the kernel evicts image pages to swap when an application
+needs memory, first note the image size with ``df -h /``:
+
+.. code-block:: shell
+
+   df -h /
+
+This shows how much tmpfs space the image occupies — that is the amount of RAM
+currently holding the node image.
+
+Now apply memory pressure using ``stress-ng`` (install it in the node image if
+not already present). The allocation must exceed **available** RAM — not just
+total RAM — to force the kernel to evict image pages. Compute the target from
+``MemTotal``:
+
+.. code-block:: shell
+
+   stress-ng --vm 1 \
+     --vm-bytes $(awk '/MemTotal/{printf "%dM", $2*0.9/1024}' /proc/meminfo) \
+     --vm-keep -t 60s &
+
+While ``stress-ng`` is running, observe memory usage:
+
+.. code-block:: shell
+
+   free -h
+
+.. code-block:: text
+
+   $ free -h
+                  total        used        free      shared  buff/cache   available
+   Mem:            15Gi        15Gi        32Mi       120Mi       512Mi       192Mi
+   Swap:           8.0Gi       4.2Gi       3.8Gi
+
+The ``Swap: used`` value has grown by roughly the size of the node image. The
+kernel has evicted cold image pages to swap, making that RAM available to the
+application. The application can access the full physical memory of the node,
+not just what is left over after the image is loaded.
+
+After ``stress-ng`` finishes, the evicted image pages remain on swap until
+they are accessed again, so ``free -h`` will continue to show swap usage until
+the node is under less pressure and pages are faulted back in as needed.
+
+Moving image pages to swap proactively
+---------------------------------------
+
+Rather than simulating a workload, you can instruct the kernel to push image
+pages to swap directly. On Linux 6.1 and later with cgroup v2, write the
+desired reclaim size to ``memory.reclaim``:
+
+.. code-block:: shell
+
+   # Request that the kernel reclaim up to 4 GiB from the root cgroup
+   # (adjust to match your image size shown by df -h /)
+   echo "4G" > /sys/fs/cgroup/memory.reclaim
+
+If the kernel cannot reclaim the full requested amount it returns an error:
+
+.. code-block:: text
+
+   -bash: echo: write error: Resource temporarily unavailable
+
+This is expected and benign. It means the kernel reclaimed as much as it could
+but the reclaimable pool was exhausted before reaching the target.
+
+The kernel will swap out reclaimable pages until the target is met or exhausted, then stop.
+Run ``free -h`` immediately afterwards to see the reduction in ``Mem: used``
+and corresponding increase in ``Swap: used``.
+
+This works best when run early in the boot process, before applications start,
+when nearly all anonymous memory belongs to the image. To automate it, add a
+systemd unit to a custom overlay that runs at ``local-fs.target``:
+
+.. code-block:: ini
+
+   [Unit]
+   Description=Reclaim node image memory to swap
+   After=local-fs.target
+   ConditionPathExists=/sys/fs/cgroup/memory.reclaim
+
+   [Service]
+   Type=oneshot
+   ExecStart=/bin/sh -c 'echo "$(df --output=used / | tail -1)K" > /sys/fs/cgroup/memory.reclaim'
+
+   [Install]
+   WantedBy=multi-user.target
+
+This reads the actual image size from ``df`` and requests that exact amount be
+reclaimed, so it adapts automatically as the image grows or shrinks.
+
 .. _provision to disk:
 
 Provision to disk

userdocs/overlays/overlays.rst

@@ -520,6 +520,10 @@ include wipefs to detect existing file systems.
 
    wwctl image exec rockylinux-9 -- /usr/bin/dracut --force --no-hostonly --add wwinit --install mkswap --regenerate-all
 
+For a complete walkthrough of configuring swap to free memory consumed by the
+in-memory node image, including which root filesystem types support swap
+reclamation, see :ref:`swap-and-image-memory`.
+
 systemd mounts
 --------------
 

userdocs/troubleshooting/troubleshooting.rst

@@ -251,6 +251,11 @@ argument:
 By default this is set to 50% of physical RAM. Note that tmpfs is required for
 SELinux overlays since initramfs cannot preserve SELinux contexts.
 
+Because the root is ``tmpfs``, the kernel can also swap cold image pages to a
+local swap device, freeing RAM for running workloads. This does not apply to the
+default ``initramfs`` root (single-stage boot), where pages are pinned in memory
+and cannot be swapped. See :ref:`swap-and-image-memory` for a complete walkthrough.
+
 .. note::
 
    On some systems, it may also be necessary to include the ``noefi`` kernel