152 lines
5.7 KiB
ReStructuredText
152 lines
5.7 KiB
ReStructuredText
============
|
|
Known issues
|
|
============
|
|
|
|
SELinux and IPMI Write not Working When Using Two-Stage Boot
|
|
============================================================
|
|
|
|
The dracut implementation of two-stage boot in versions of Warewulf prior to
|
|
v4.6.0 bypasses the ``wwinit`` process by default, invoking the image's init
|
|
system directly. While cluster nodes will often still boot mostly successfully
|
|
this way, features implemented by wwinit will not complete. In particular,
|
|
SELinux relabeling and IPMI write are not executed.
|
|
|
|
To ensure that dracut runs the full ``wwinit`` process, pass ``init=/init`` or
|
|
``init=/warewulf/wwinit`` on the kernel command line.
|
|
|
|
.. code-block:: shell
|
|
|
|
wwctl profile set default --kernelargs="init=/init"
|
|
|
|
Images are Read-Only
|
|
====================
|
|
|
|
Warewulf v4.5 uses the permissions on an image's ``rootfs/`` to determine a
|
|
"read-only" state of the image: if the root directory of the image is ``u-w``,
|
|
it will be mounted read-only during ``wwctl image <exec|shell``, preventing
|
|
interactive changes to the image.
|
|
|
|
In the past, the root directory was ``u+w``, but Enterprise Linux 9.5 (including
|
|
Red Hat, Rocky, _et al._) includes an update to the ``filesystem`` package that
|
|
marks the root directory ``u-w``. This causes Warewulf images to be "read only"
|
|
by default.
|
|
|
|
To mark a Warewulf image as writeable, use `chmod u+w`.
|
|
|
|
.. code-block:: shell
|
|
|
|
chmod u+w $(wwctl image show rockylinux-9.5)
|
|
|
|
This behavior is changed in v4.6 to use an explicit ``readonly`` file stored
|
|
outside of ``rootfs/``.
|
|
|
|
Image Sockets Cause Build Failures
|
|
==================================
|
|
|
|
If an image source directory includes persistent sockets, these sockets may
|
|
cause the import operation to fail.
|
|
|
|
.. code-block::
|
|
|
|
Copying sources...
|
|
ERROR : could not import image: lchown ./rockylinux-8/run/user/0/gnupg/d.kg8ijih5tq41ixoeag4p1qup/S.gpg-agent: no such file or directory
|
|
|
|
To resolve this, remove the sockets from the source directory.
|
|
|
|
.. code-block:: shell
|
|
|
|
find ./rockylinux-8/ -type s -delete
|
|
|
|
This issue was fixed in an upstream library and `should be resolved in Warewulf
|
|
v4.6.0. <https://github.com/warewulf/warewulf/issues/892>`_
|
|
|
|
Image Size Considerations
|
|
=========================
|
|
|
|
Node images can grow quickly as packages and other files are added to them. Even
|
|
these larger images are often not an issue in modern environments; but some
|
|
architectural limits exist that can impede the use of images larger than a few
|
|
gigabytes. Workarounds exist for these issues in most circumstances:
|
|
|
|
* Warewulf's :ref:`two-stage boot support <booting with dracut>` effectively
|
|
eliminates this problem by handling the bulk of the image management within
|
|
Linux. This feature is currently in preview, and is subject to change; but it
|
|
is likely to become the default boot method in a future release.
|
|
|
|
* Systems booting in legacy / BIOS mode, being a 32-bit environment, cannot boot
|
|
an image that requires more than 4GB to decompress. This means that the
|
|
compressed image and the decompressed image together must be < 4GB. This is
|
|
typically reported by the system as "No space left on device
|
|
(https://ipxe.org/34182006)."
|
|
|
|
The best work-around for this limitation is to switch to UEFI. UEFI is 64-bit
|
|
and should support booting significantly larger images, though sometimes
|
|
system-specific implementation details have led to artificial limitations on
|
|
image size.
|
|
|
|
* The Linux kernel itself can only decompress an image up to 4GB due to the use
|
|
of 32-bit integers in critical sections of the kernel initrd decompression
|
|
code.
|
|
|
|
The best work-around for this limitation is to use an iPXE with support for
|
|
`imgextract <https://ipxe.org/cmd/imgextract>`_. This allows iPXE to
|
|
decompress the image rather than the kernel.
|
|
|
|
* Some BIOS / firmware retain a "memory hole" feature for legacy devices, e.g.,
|
|
reserving a 1MB block of memory at the 15MB-16MB address range. This feature
|
|
can interfere with booting stateless node images.
|
|
|
|
If you are still getting "Not enough memory" or "No space left on device"
|
|
errors, try disabling any "memory hole" features or updating your system BIOS
|
|
or firmware.
|
|
|
|
.. _arp-cache-overflow-on-large-clusters:
|
|
|
|
ARP Cache Overflow on Large Clusters
|
|
=====================================
|
|
|
|
On clusters with many nodes, the Linux kernel's default ARP cache limits may be
|
|
too low, causing the Warewulf server to log:
|
|
|
|
.. code-block:: text
|
|
|
|
neighbour: arp_cache: neighbor table overflow!
|
|
|
|
The kernel manages the ARP cache with three garbage-collection thresholds:
|
|
|
|
* ``net.ipv4.neigh.default.gc_thresh1`` -- no garbage collection below this
|
|
count (default: 128)
|
|
* ``net.ipv4.neigh.default.gc_thresh2`` -- garbage collection is triggered above
|
|
this count (default: 512)
|
|
* ``net.ipv4.neigh.default.gc_thresh3`` -- hard limit; entries are dropped above
|
|
this count (default: 1024)
|
|
|
|
This is particularly relevant for Warewulf because ``warewulfd`` identifies nodes
|
|
by looking up their IP address in the kernel's ARP cache (``/proc/net/arp``) during
|
|
provisioning. If cache entries for cluster nodes are evicted, node identification
|
|
can fail.
|
|
|
|
Increase the thresholds on the Warewulf server. As a starting point:
|
|
|
|
.. code-block:: shell
|
|
|
|
sysctl -w net.ipv4.neigh.default.gc_thresh2=1024
|
|
sysctl -w net.ipv4.neigh.default.gc_thresh3=2048
|
|
|
|
For larger clusters (hundreds of nodes or more), higher values may be needed:
|
|
|
|
.. code-block:: shell
|
|
|
|
sysctl -w net.ipv4.neigh.default.gc_thresh2=2048
|
|
sysctl -w net.ipv4.neigh.default.gc_thresh3=4096
|
|
|
|
To make the change persistent across reboots, create a file in ``/etc/sysctl.d/``:
|
|
|
|
.. code-block:: shell
|
|
|
|
cat > /etc/sysctl.d/90-warewulf-arp.conf << 'EOF'
|
|
net.ipv4.neigh.default.gc_thresh2 = 2048
|
|
net.ipv4.neigh.default.gc_thresh3 = 4096
|
|
EOF
|
|
sysctl -p /etc/sysctl.d/90-warewulf-arp.conf
|