From 3f12d5635c7037e2ce6edd7fab4c6e67c4a08d48 Mon Sep 17 00:00:00 2001
From: Jonathon Anderson <janderson@ciq.com>
Date: Wed, 11 Mar 2026 23:22:07 -0600
Subject: [PATCH] Document configuring the arp cache for large clusters

Signed-off-by: Jonathon Anderson <janderson@ciq.com>
---
 userdocs/server/routes.rst                |  6 +++
 userdocs/troubleshooting/known-issues.rst | 50 +++++++++++++++++++++++
 2 files changed, 56 insertions(+)

diff --git a/userdocs/server/routes.rst b/userdocs/server/routes.rst
index 9d75ec0f..265f13a9 100644
--- a/userdocs/server/routes.rst
+++ b/userdocs/server/routes.rst
@@ -181,6 +181,12 @@ of the client's IP address against the kernel's ARP cache (``/proc/net/arp``).
 This route is intended for EFI HTTP Boot clients, where the firmware fetches
 a boot URI from DHCP and cannot perform variable substitution.
 
+.. note::
+
+   On large clusters the kernel's default ARP cache limits may be exceeded,
+   causing node identification to fail. See
+   :ref:`arp-cache-overflow-on-large-clusters` for tuning guidance.
+
 The ``{file}`` component determines what is served:
 
 * ``shim.efi``: Serves the ``shim.efi`` binary extracted from the node's
diff --git a/userdocs/troubleshooting/known-issues.rst b/userdocs/troubleshooting/known-issues.rst
index c0fa97c2..51cac30b 100644
--- a/userdocs/troubleshooting/known-issues.rst
+++ b/userdocs/troubleshooting/known-issues.rst
@@ -99,3 +99,53 @@ gigabytes. Workarounds exist for these issues in most circumstances:
   If you are still getting "Not enough memory" or "No space left on device"
   errors, try disabling any "memory hole" features or updating your system BIOS
   or firmware.
+
+.. _arp-cache-overflow-on-large-clusters:
+
+ARP Cache Overflow on Large Clusters
+=====================================
+
+On clusters with many nodes, the Linux kernel's default ARP cache limits may be
+too low, causing the Warewulf server to log:
+
+.. code-block:: text
+
+   neighbour: arp_cache: neighbor table overflow!
+
+The kernel manages the ARP cache with three garbage-collection thresholds:
+
+* ``net.ipv4.neigh.default.gc_thresh1`` -- no garbage collection below this
+  count (default: 128)
+* ``net.ipv4.neigh.default.gc_thresh2`` -- garbage collection is triggered above
+  this count (default: 512)
+* ``net.ipv4.neigh.default.gc_thresh3`` -- hard limit; entries are dropped above
+  this count (default: 1024)
+
+This is particularly relevant for Warewulf because ``warewulfd`` identifies nodes
+by looking up their IP address in the kernel's ARP cache (``/proc/net/arp``) during
+provisioning. If cache entries for cluster nodes are evicted, node identification
+can fail.
+
+Increase the thresholds on the Warewulf server. As a starting point:
+
+.. code-block:: shell
+
+   sysctl -w net.ipv4.neigh.default.gc_thresh2=1024
+   sysctl -w net.ipv4.neigh.default.gc_thresh3=2048
+
+For larger clusters (hundreds of nodes or more), higher values may be needed:
+
+.. code-block:: shell
+
+   sysctl -w net.ipv4.neigh.default.gc_thresh2=2048
+   sysctl -w net.ipv4.neigh.default.gc_thresh3=4096
+
+To make the change persistent across reboots, create a file in ``/etc/sysctl.d/``:
+
+.. code-block:: shell
+
+   cat > /etc/sysctl.d/90-warewulf-arp.conf << 'EOF'
+   net.ipv4.neigh.default.gc_thresh2 = 2048
+   net.ipv4.neigh.default.gc_thresh3 = 4096
+   EOF
+   sysctl -p /etc/sysctl.d/90-warewulf-arp.conf