kelvin/warewulf
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Document configuring the arp cache for large clusters

Browse Source 
Signed-off-by: Jonathon Anderson <janderson@ciq.com>
This commit is contained in:
Jonathon Anderson
2026-03-11 23:22:07 -06:00
parent fe8f4b5b11
commit 3f12d5635c
2 changed files with 56 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
userdocs/troubleshooting/known-issues.rst
View File

@@ -99,3 +99,53 @@ gigabytes. Workarounds exist for these issues in most circumstances:
If you are still getting "Not enough memory" or "No space left on device"
errors, try disabling any "memory hole" features or updating your system BIOS
or firmware.
.. _arp-cache-overflow-on-large-clusters:
ARP Cache Overflow on Large Clusters
=====================================
On clusters with many nodes, the Linux kernel's default ARP cache limits may be
too low, causing the Warewulf server to log:
.. code-block:: text
neighbour: arp_cache: neighbor table overflow!
The kernel manages the ARP cache with three garbage-collection thresholds:
* ``net.ipv4.neigh.default.gc_thresh1`` -- no garbage collection below this
count (default: 128)
* ``net.ipv4.neigh.default.gc_thresh2`` -- garbage collection is triggered above
this count (default: 512)
* ``net.ipv4.neigh.default.gc_thresh3`` -- hard limit; entries are dropped above
this count (default: 1024)
This is particularly relevant for Warewulf because ``warewulfd`` identifies nodes
by looking up their IP address in the kernel's ARP cache (``/proc/net/arp``) during
provisioning. If cache entries for cluster nodes are evicted, node identification
can fail.
Increase the thresholds on the Warewulf server. As a starting point:
.. code-block:: shell
sysctl -w net.ipv4.neigh.default.gc_thresh2=1024
sysctl -w net.ipv4.neigh.default.gc_thresh3=2048
For larger clusters (hundreds of nodes or more), higher values may be needed:
.. code-block:: shell
sysctl -w net.ipv4.neigh.default.gc_thresh2=2048
sysctl -w net.ipv4.neigh.default.gc_thresh3=4096
To make the change persistent across reboots, create a file in ``/etc/sysctl.d/``:
.. code-block:: shell
cat > /etc/sysctl.d/90-warewulf-arp.conf << 'EOF'
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096
EOF
sysctl -p /etc/sysctl.d/90-warewulf-arp.conf