kelvin/warewulf
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Documentation reorg for v4.6.0

Browse Source 
Signed-off-by: Jonathon Anderson <janderson@ciq.com>
This commit is contained in:
Jonathon Anderson
2025-02-18 16:26:05 -07:00
parent 98695cbbff
commit 9e41378e1c
65 changed files with 3493 additions and 4654 deletions
Download Patch File Download Diff File Expand all files Collapse all files

101
userdocs/troubleshooting/known-issues.rst Normal file
View File

@@ -0,0 +1,101 @@
============
Known issues
============
SELinux and IPMI Write not Working When Using Two-Stage Boot
============================================================
The dracut implementation of two-stage boot in versions of Warewulf prior to
v4.6.0 bypasses the ``wwinit`` process by default, invoking the image's init
system directly. While cluster nodes will often still boot mostly successfully
this way, features implemented by wwinit will not complete. In particular,
SELinux relabeling and IPMI write are not executed.
To ensure that dracut runs the full ``wwinit`` process, pass ``init=/init`` or
``init=/warewulf/wwinit`` on the kernel command line.
.. code-block:: shell
wwctl profile set default --kernelargs="init=/init"
Images are Read-Only
====================
Warewulf v4.5 uses the permissions on an image's ``rootfs/`` to determine a
"read-only" state of the image: if the root directory of the image is ``u-w``,
it will be mounted read-only during ``wwctl image <exec|shell``, preventing
interactive changes to the image.
In the past, the root directory was ``u+w``, but Enterprise Linux 9.5 (including
Red Hat, Rocky, _et al._) includes an update to the ``filesystem`` package that
marks the root directory ``u-w``. This causes Warewulf images to be "read only"
by default.
To mark a Warewulf image as writeable, use `chmod u+w`.
.. code-block:: shell
chmod u+w $(wwctl image show rockylinux-9.5)
This behavior is changed in v4.6 to use an explicit ``readonly`` file stored
outside of ``rootfs/``.
Image Sockets Cause Build Failures
==================================
If an image source directory includes persistent sockets, these sockets may
cause the import operation to fail.
.. code-block::
Copying sources...
ERROR : could not import image: lchown ./rockylinux-8/run/user/0/gnupg/d.kg8ijih5tq41ixoeag4p1qup/S.gpg-agent: no such file or directory
To resolve this, remove the sockets from the source directory.
.. code-block:: shell
find ./rockylinux-8/ -type s -delete
This issue was fixed in an upstream library and `should be resolved in Warewulf
v4.6.0. <https://github.com/warewulf/warewulf/issues/892>`_
Image Size Considerations
=========================
Node images can grow quickly as packages and other files are added to them. Even
these larger images are often not an issue in modern environments; but some
architectural limits exist that can impede the use of images larger than a few
gigabytes. Workarounds exist for these issues in most circumstances:
* Warewulf's :ref:`two-stage boot support <booting with dracut>` effectively
eliminates this problem by handling the bulk of the image management within
Linux. This feature is currently in preview, and is subject to change; but it
is likely to become the default boot method in a future release.
* Systems booting in legacy / BIOS mode, being a 32-bit environment, cannot boot
an image that requires more than 4GB to decompress. This means that the
compressed image and the decompressed image together must be < 4GB. This is
typically reported by the system as "No space left on device
(https://ipxe.org/34182006)."
The best work-around for this limitation is to switch to UEFI. UEFI is 64-bit
and should support booting significantly larger images, though sometimes
system-specific implementation details have led to artificial limitations on
image size.
* The Linux kernel itself can only decompress an image up to 4GB due to the use
of 32-bit integers in critical sections of the kernel initrd decompression
code.
The best work-around for this limitation is to use an iPXE with support for
`imgextract <https://ipxe.org/cmd/imgextract>`_. This allows iPXE to
decompress the image rather than the kernel.
* Some BIOS / firmware retain a "memory hole" feature for legacy devices, e.g.,
reserving a 1MB block of memory at the 15MB-16MB address range. this feature
can interfere with booting stateless node images.
If you are still getting "Not enough memory" or "No space left on device"
errors, try disabling any "memory hole" features or updating your system BIOS
or firmware.

188
userdocs/troubleshooting/troubleshooting.rst Normal file
View File

@@ -0,0 +1,188 @@
===============
Troubleshooting
===============
warewulfd
=========
The Warewulf server (``warewulfd``) sends logs to the systemd journal.
.. code-block::
journalctl -u warewulfd.service
To increase the verbosity of the log, specify either ``--verbose`` or
``--debug`` in the warewulfd OPTIONS.
.. code-block::
echo "OPTIONS=--debug" >>/etc/default/warewulfd
systemctl restart warewulfd.service
iPXE
====
If you're using iPXE to boot (the default), you can get a command prompt by
pressing with C-b during boot.
From the iPXE command prompt, you can run the same commands from `default.ipxe`_
to troubleshoot potential boot problems.
.. _default.ipxe: https://github.com/warewulf/warewulf/blob/main/etc/ipxe/default.ipxe
For example, the following commands perform a (relatively) normal Warewulf boot.
(Substitute your Warewulf server's IP address in place of 10.0.0.1, update the
port number if you have changed it from the default of 9873, and substitute your
cluster node's MAC address in place of 00:00:00:00:00:00.)
.. code-block::
set uri http://10.0.0.1:9873/provision/00:00:00:00:00:00
kernel --name kernel ${uri}?stage=kernel
imgextract --name image ${uri}?stage=image&compress=gz
imgextract --name system ${uri}?stage=system&compress=gz
imgextract --name runtime ${uri}?stage=runtime&compress=gz
boot kernel initrd=image initrd=system initrd=runtime
- The ``uri`` variable points to ``warewulfd`` for future reference. This
includes the cluster node's MAC address so that Warewulf knows what image and
overlays to provide.
- The ``kernel`` command fetches a kernel for later booting.
- The ``imgextract`` command fetches and decompresses the images that will make
up the booted noe image. In a typical environment this is used to load a
minimal "initial ramdisk" which, then, boots the rest of the system. Warewulf,
by default, loads the entire image as an initial ramdisk, and also loads the
system and runtime overlays at this time time.
- The ``boot`` command tells iPXE to boot the system with the given kernel and
ramdisks.
.. note::
This example does not provide ``assetkey`` information to ``warewulfd``. If
your nodes have defined asset tags, provide it in the ``uri`` variable for
the node you are trying to boot.
For example, you may want to try booting to a pre-init shell with debug logging
enabled. To do so, substitute the ``boot`` command above.
.. code-block::
boot kernel initrd=image initrd=system initrd=runtime rdinit=/bin/sh
.. note::
You may be more familiar with specifying ``init=`` on the kernel command
line. ``rdinit`` indicates "ramdisk init." Since Warewulf, by default, boots
the node image as an initial ramdisk, we must use ``rdinit=`` here.
GRUB
====
If you're using GRUB to boot, you can get a command prompt by pressing "c" when
prompted during boot.
From the GRUB command prompt, you can enter the same commands that you would
otherwise find in `grub.cfg.ww`_.
.. _grub.cfg.ww: https://github.com/warewulf/warewulf/blob/main/etc/grub/grub.cfg.ww
For example, the following commands perform a (relatively) normal Warewulf boot.
(Substitute your Warewulf server's IP address in place of 10.0.0.1, and update
the port number if you have changed it from the default of 9873.)
.. code-block::
uri="(http,10.0.0.1:9873)/provision/${net_default_mac}"
linux "${uri}?stage=kernel" wwid=${net_default_mac}
initrd "${uri}?stage=image&compress=gz" "${uri}?stage=system&compress=gz" "${uri}?stage=runtime&compress=gz"
boot
- The ``uri`` variable points to ``warewulfd`` for future reference.
``${net_default_mac}`` provides Warewulf with the MAC address of the booting
node, so that Warewulf knows what image and overlays to provide it.
- The ``linux`` command tells GRUB what kernel to boot, as provided by
``warewulfd``. The ``wwid`` kernel argument helps ``wwclient`` identify the
node during runtime.
- The ``initrd`` command tells GRUB what images to load into memory for boot. In
a typical environment this is used to load a minimal "initial ramdisk" which,
then, boots the rest of the system. Warewulf, by default, loads the entire
image as an initial ramdisk, and also loads the system and runtime overlays at
this time time.
- The ``boot`` command tells GRUB to boot the system with the previously-defined
configuration.
.. note::
This example does not provide ``assetkey`` information to ``warewulfd``. If
your nodes have defined asset tags, provide it in the ``uri`` variable for
the node you are trying to boot.
For example, you may want to try booting to a pre-init shell with debug logging
enabled. To do so, substitute the ``linux`` command above.
.. code-block::
linux "${uri}?stage=kernel" wwid=${net_default_mac} debug rdinit=/bin/sh
.. note::
You may be more familiar with specifying ``init=`` on the kernel command
line. ``rdinit`` indicates "ramdisk init." Since Warewulf, by default, boots
the node image as an initial ramdisk, we must use ``rdinit=`` here.
Dracut
======
By default, dracut simply panics and terminates when it encounters an issue.
Dracut looks at the kernel command line for its configuration. You can configure
it for additional logging and to switch to an interactive shell on error:
.. code-block::
wwctl profile set default --kernelargs=rd.shell,rd.debug,log_buf_len=1M
For more information on debugging Dracut problems, see `the Fedora dracut
problems guide.`_
.. _the Fedora dracut problems guide.: https://docs.fedoraproject.org/en-US/quick-docs/debug-dracut-problems/
Ignition
========
If partition creation doesn't work as expected you have a few options to
investigate:
- Add ``systemd.log_level=debug`` and or ``rd.debug`` to the kernelArgs of the
node you're working on.
- After the next boot you should be able to find verbose information on the node
with ``journalctl -u ignition-ww4-disks.service``.
- You could also check the content of ``/warewulf/ignition.json``.
- You could try to tinker with ``/warewulf/ignition.json`` calling
.. code-block:: shell
/usr/lib/dracut/modules.d/30ignition/ignition \
--platform=metal \
--stage=disks \
--config-cache=/warewulf/ignition.json \
--log-to-stdout
after each iteration on the node directly until you find the settings you
need. (Make sure to unmount all partitions if ``ignition`` was partially
successful.)
- Sometimes you need to add ``should_exist: "true"`` for the swap partition as
well.
Running Containers on Cluster Nodes
===================================
Some container runtimes, notably Podman, require file system features that are
not available in ``initrootfs``. Cluster nodes using Podman (and some other
container runtimes) should be configured with ``--root=tmpfs``.