422 lines
15 KiB
ReStructuredText
422 lines
15 KiB
ReStructuredText
====================
|
|
Container Management
|
|
====================
|
|
|
|
Since the inception of Warewulf over 20 years ago, Warewulf has used
|
|
the model of the "Virtual Node File System" (VNFS) as a template image
|
|
for the compute nodes. This is similar to a golden master image,
|
|
except that the node file system exists within a directory on the
|
|
Warewulf control node (e.g. a ``chroot()``).
|
|
|
|
In hindsight, we've been using containers all along, but the buzzword
|
|
just didn't exist. Over the last 5-6 years, the enterprise has created
|
|
a lot of tooling and standards around defining, building,
|
|
distributing, securing, and managing containers, so Warewulf v4 now
|
|
integrates directly within the container ecosystem to facilitate the
|
|
process of VNFS image management.
|
|
|
|
If you are not currently leveraging the container ecosystem in any
|
|
other way, you can still build your own chroot directories and use
|
|
Warewulf as before.
|
|
|
|
It is important to understand that Warewulf is not running a container
|
|
runtime on cluster nodes. While it is absolutely possible to run
|
|
containers on cluster nodes, Warewulf is provisioning the container
|
|
image to the bare metal and booting it. This container will be used as
|
|
the base operating system and, by default, it will run entirely in
|
|
memory. This means that when you reboot the node, the node retains no
|
|
information about Warewulf or how it booted.
|
|
|
|
Container Tools
|
|
===============
|
|
|
|
There are different container managment tools available. Docker is
|
|
probably the most recognizable one in the enterprise. Podman is
|
|
another one that is gaining traction on the RHEL platforms. In HPC,
|
|
Apptainer is the most utilized container management tool. You can use
|
|
any of these to create and manage the containers to be later imported
|
|
into Warewulf.
|
|
|
|
Importing Containers
|
|
====================
|
|
|
|
Warewulf supports importing an image from any OCI compliant
|
|
registry. This means you can import from a public registry or from a
|
|
private registry.
|
|
|
|
Here is an example of importing from Docker Hub.
|
|
|
|
.. code-block:: console
|
|
|
|
# wwctl container import docker://ghcr.io/warewulf/warewulf-rockylinux:8 rocky-8
|
|
Getting image source signatures
|
|
Copying blob d7f16ed6f451 done
|
|
Copying config da2ca70704 done
|
|
Writing manifest to image destination
|
|
Storing signatures
|
|
[LOG] info unpack layer: sha256:d7f16ed6f45129c7f4adb3773412def4ba2bf9902de42e86e77379a65d90a984
|
|
Updating the container's /etc/resolv.conf
|
|
Building container: rocky-8
|
|
|
|
.. note::
|
|
|
|
Most containers in Docker Hub are not "bootable", in that, they
|
|
have a limited version of Systemd to make them lighter weight for
|
|
container purposes. For this reason, don't expect any base Docker
|
|
container (e.g. ``docker://rockylinux`` or ``docker://debian``) to
|
|
boot properly. They will not, as they will get stuck into a single
|
|
user mode. The containers in `https://hub.docker.com/u/warewulf
|
|
<https://hub.docker.com/u/warewulf>`_ are not limited and thus
|
|
they boot as you would expect.
|
|
|
|
Private Registry
|
|
----------------
|
|
|
|
It is possible to use a private registry that is password protected or
|
|
does not have the requirement for TLS. In order to do so, you have two
|
|
choices for handling the credentials.
|
|
|
|
* Set environmental variables
|
|
* Use ``docker login`` or ``podman login`` which will store the
|
|
credentials locally
|
|
|
|
Please note, there is no requirement to install and use docker or
|
|
podman on your control node just for importing images into Warewulf.
|
|
|
|
Here are the environmental variables that can be used.
|
|
|
|
.. code-block:: console
|
|
|
|
WAREWULF_OCI_USERNAME
|
|
WAREWULF_OCI_PASSWORD
|
|
WAREWULF_OCI_NOHTTPS
|
|
|
|
They can be overwritten with ``--nohttps``, ``--username`` and ``--password``.
|
|
.. code-block:: console
|
|
|
|
# wwctl import --username tux --password supersecret docker://ghcr.io/privatereg/rocky:8
|
|
|
|
The above is just an example. Consideration should be done before
|
|
doing it this way if you are in a security sensitive environment or
|
|
shared environments as this command line wil show up in the process
|
|
table.
|
|
|
|
Local Files
|
|
-----------
|
|
|
|
It is also possible to import a container from a local file or
|
|
directory. For example, Podman can save a `.tar` archive of an OCI
|
|
image. This archive can be directly imported into Warewulf, no
|
|
registry required.
|
|
|
|
.. code-block:: console
|
|
|
|
# podman save alpine:latest >alpine.tar
|
|
# wwctl container import alpine.tar alpine
|
|
|
|
Chroot directories and Apptainer sandbox images can also be imported
|
|
directly.
|
|
|
|
.. code-block:: console
|
|
|
|
$ apptainer build --sandbox ./rockylinux-8/ docker://ghcr.io/warewulf/warewulf-rockylinux:8
|
|
$ sudo wwctl container import ./rockylinux-8/ rockylinux-8
|
|
|
|
HTTP proxies
|
|
------------
|
|
|
|
You can set ``HTTP_PROXY``, ``HTTPS_PROXY``, and ``NO_PROXY`` (or their
|
|
lower-case versions) to use a proxy during ``wwctl container import``.
|
|
|
|
.. code-block:: shell
|
|
|
|
export HTTPS_PROXY=squid.localdomain
|
|
wwctl conatiner import docker://ghcr.io/warewulf/warewulf-rockylinux:8
|
|
|
|
See ProxyFromEnvironment_ For more information.
|
|
|
|
.. _ProxyFromEnvironment: https://pkg.go.dev/net/http#ProxyFromEnvironment
|
|
|
|
.. note::
|
|
|
|
OCI and ORAS registries typically use HTTPS, so you probably need to set
|
|
``HTTPS_PROXY`` or ``https_proxy`` rather than the ``HTTP`` variants.
|
|
|
|
Syncuser
|
|
========
|
|
|
|
At import time Warewulf checks if the names of the users on the host
|
|
match the users and UIDs/GIDs in the imported container. If there is
|
|
mismatch, the import command will print out a warning. By setting the
|
|
``--syncuser`` flag you advise Warewulf to try to syncronize the users
|
|
from the host to the container, which means that ``/etc/passwd`` and
|
|
``/etc/group`` of the imported container are updated and all the files
|
|
belonging to these UIDs and GIDs will also be updated.
|
|
|
|
A check if the users of the host and container matches can be
|
|
triggered with the ``syncuser`` command.
|
|
|
|
.. code-block:: console
|
|
|
|
# wwctl container syncuser container-name
|
|
|
|
With the ``--write`` flag it will update the container to match the
|
|
user database of the host as described above.
|
|
|
|
.. code-block:: console
|
|
|
|
wwctl container syncuser --write container-name
|
|
|
|
Listing All Imported Containers
|
|
===============================
|
|
|
|
Once the container has been imported, you can list them all with the
|
|
following command:
|
|
|
|
.. code-block:: console
|
|
|
|
# wwctl container list
|
|
CONTAINER NAME BUILT NODES
|
|
rocky-8 true 0
|
|
|
|
Once a container has been imported and showing up in this list you can
|
|
configure it to boot compute nodes.
|
|
|
|
Making Changes To Containers
|
|
============================
|
|
|
|
Warewulf has a minimal container runtime built into it. This means you
|
|
can run commands inside of any of the containers and make changes to
|
|
them as follows:
|
|
|
|
.. code-block:: console
|
|
|
|
# wwctl container exec rocky-8 /bin/sh
|
|
[rocky-8] Warewulf> cat /etc/rocky-release
|
|
Rocky Linux release 8.4 (Green Obsidian)
|
|
[rocky-8] Warewulf> exit
|
|
Rebuilding container...
|
|
[INFO] Skipping (VNFS is current)
|
|
|
|
You can also ``--bind`` directories from your host into the container
|
|
when using the exec command. This works as follows:
|
|
|
|
.. code-block:: console
|
|
|
|
# wwctl container exec --bind /tmp:/mnt rocky-8 /bin/sh
|
|
[rocky-8] Warewulf>
|
|
|
|
.. note::
|
|
|
|
As with any mount command, both the source and the target must
|
|
exist. This is why the example uses the ``/mnt/`` directory
|
|
location, as it is almost always present and empty in every Linux
|
|
distribution (as prescribed by the LSB file hierarchy standard).
|
|
|
|
Files which should always be present in a container image like ``resolv.conf``
|
|
can be specified in ``warewulf.conf``:
|
|
|
|
.. code-block:: yaml
|
|
container mounts:
|
|
- source: /etc/resolv.conf
|
|
dest: /etc/resolv.conf
|
|
readonly: true
|
|
|
|
.. note::
|
|
Instead of ``readonly: true`` you can set ``copy: true``. This causes the
|
|
source file to be copied to the container and removed if it was not
|
|
modified. This can be useful for files used for registrations.
|
|
|
|
When the command completes, if anything within the container changed,
|
|
the container will be rebuilt into a bootable static object
|
|
automatically.
|
|
|
|
If the files ``/etc/passwd`` or ``/etc/group`` were updated, there
|
|
will be an additional check to confirm if the users are in sync as
|
|
described in `Syncuser`_ section.
|
|
|
|
Excluding Files from a Container
|
|
--------------------------------
|
|
|
|
Warewulf can exclude files from a source container to prevent them
|
|
from being delivered to the compute node. This is typically used to
|
|
reduce the size of the image when some files are unnecessary.
|
|
|
|
Patterns for excluded files are read from the file
|
|
``/etc/warewulf/excludes`` in the container image itself. For example,
|
|
the default Rocky Linux images exclude these paths:
|
|
|
|
.. code-block::
|
|
|
|
/boot/
|
|
/usr/share/GeoIP
|
|
|
|
``/etc/warewulf/excludes`` supports the patterns implemented by
|
|
`filepath.Match <https://pkg.go.dev/path/filepath#Match>`_.
|
|
|
|
Preparing a container for build
|
|
-------------------------------
|
|
|
|
Warewulf executes the script ``/etc/warewulf/container_exit.sh`` after
|
|
a ``wwctl container shell`` or ``wwctl container exec`` and prior to
|
|
(re)building the final node image for delivery. This is typically used
|
|
to remove cache or log files that may have been generated by the
|
|
executed command or interactive session.
|
|
|
|
For example, the default Rocky Linux images runs ``dnf clean all`` to
|
|
remove any package repository caches that may have been generated.
|
|
|
|
Creating Containers From Scratch
|
|
================================
|
|
|
|
You can also create containers from scratch and import those
|
|
containers into Warewulf as previous versions of Warewulf did.
|
|
|
|
Building A Container From Your Host
|
|
-----------------------------------
|
|
|
|
RPM based distributions, as well as Debian variants can all bootstrap
|
|
mini ``chroot()`` directories which can then be used to bootstrap your
|
|
node's container.
|
|
|
|
For example, on an RPM based Linux distribution with YUM or DNF, you
|
|
can do something like the following:
|
|
|
|
.. code-block:: console
|
|
|
|
# yum install --installroot /tmp/newroot basesystem bash \
|
|
chkconfig coreutils e2fsprogs ethtool filesystem findutils \
|
|
gawk grep initscripts iproute iputils net-tools nfs-utils pam \
|
|
psmisc rsync sed setup shadow-utils rsyslog tzdata util-linux \
|
|
words zlib tar less gzip which util-linux openssh-clients \
|
|
openssh-server dhclient pciutils vim-minimal shadow-utils \
|
|
strace cronie crontabs cpio wget rocky-release ipmitool yum \
|
|
NetworkManager
|
|
|
|
You can do something similar with Debian-based distributions:
|
|
|
|
.. code-block:: console
|
|
|
|
# apt-get install debootstrap
|
|
# debootstrap stable /tmp/newroot http://ftp.us.debian.org/debian
|
|
|
|
Once you have created and modified your new ``chroot()``, you can
|
|
import it into Warewulf with the following command:
|
|
|
|
.. code-block:: console
|
|
|
|
# wwctl container import /tmp/newroot containername
|
|
|
|
Building A Container Using Apptainer
|
|
------------------------------------
|
|
|
|
Apptainer, a container platform for HPC and performance intensive
|
|
applications, can also be used to create node containers for
|
|
Warewulf. There are several Apptainer container recipes in the
|
|
``containers/Apptainer/`` directory and can be found on GitHub at
|
|
`https://github.com/warewulf/warewulf/tree/main/containers/Apptainer
|
|
<https://github.com/warewulf/warewulf/tree/main/containers/Apptainer>`_.
|
|
|
|
You can use these as starting points and adding any additional steps
|
|
you want in the ``%post`` section of the recipe file. Once you've done
|
|
that, installing Apptainer, building a container sandbox and importing
|
|
into Warewulf can be done with the following steps:
|
|
|
|
.. code-block:: console
|
|
|
|
# yum install epel-release
|
|
# yum install Apptainer
|
|
# Apptainer build --sandbox /tmp/newroot /path/to/Apptainer/recipe.def
|
|
# wwctl container import /tmp/newroot containername
|
|
|
|
Building A Container Using Podman
|
|
---------------------------------
|
|
|
|
You can also build a container using podman via a ``Dockerfile``. For
|
|
this step the container must be exported to a tar archive, which then
|
|
can be imported to Warewulf. The following steps will create an
|
|
openSUSE Leap container and import it to Warewulf:
|
|
|
|
.. code-block:: console
|
|
|
|
# podman build -f containers/Docker/openSUSE/Containerfile --tag leap-ww
|
|
# podman save localhost/leap-ww:latest -o ~/leap-ww.tar
|
|
# wwctl container import file://root/leap-ww.tar leap-ww
|
|
|
|
Container Size Considerations
|
|
=============================
|
|
|
|
Base compute node container images start quite small (a few hundred
|
|
megabytes), but can grow quickly as packages and other files are added
|
|
to them. Even these larger images are typically not an issue in modern
|
|
environments; but some architectural limits exist that can impede the
|
|
use of images larger than a few gigabytes. Workarounds exist for these
|
|
issues in most circumstances:
|
|
|
|
* Systems booting in legacy / BIOS mode, being a 32-bit environment,
|
|
cannot boot an image that requires more than 4GB to decompress. This
|
|
means that the compressed image and the decompressed image together
|
|
must be < 4GB. This is typically reported by the system as "No space
|
|
left on device (https://ipxe.org/34182006)."
|
|
|
|
The best work-around for this limitation is to switch to UEFI. UEFI
|
|
is 64-bit and should support booting significantly larger images,
|
|
though sometimes system-specific implementation details have led to
|
|
artificial limitations on image size.
|
|
|
|
* The Linux kernel itself can only decompress an image up to 4GB due
|
|
to the use of 32-bit integers in critical sections of the kernel
|
|
initrd decompression code.
|
|
|
|
The best work-around for this limitation is to use an iPXE with
|
|
support for `imgextract <https://ipxe.org/cmd/imgextract>`_. This
|
|
allows iPXE to decompress the image rather than the kernel.
|
|
|
|
* Some BIOS / firmware retain a "memory hole" feature for legacy
|
|
devices, e.g., reserving a 1MB block of memory at the 15MB-16MB
|
|
address range. this feature can interfere with booting stateless
|
|
node images.
|
|
|
|
If you are still getting "Not enough memory" or "No space left on
|
|
device" errors, try disabling any "memory hole" features or updating
|
|
your system BIOS or firmware.
|
|
|
|
Duplicating a container
|
|
============================
|
|
It is possible to duplicate an installed image by using :
|
|
|
|
.. code-block:: console
|
|
|
|
# wwctl container copy CONTAINER_NAME DUPLICATED_CONTAINER_NAME
|
|
|
|
This kind of duplication can be useful if you are looking for canary tests.
|
|
|
|
Multi-arch container management
|
|
============================
|
|
It is possible to build, edit, and provision containers of different
|
|
architectures (i.e. aarch64) from an x86_64 host by using QEMU. Simply
|
|
run the appropriate command below based on your container management tools.
|
|
|
|
.. code-block:: console
|
|
|
|
# sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
|
|
# sudo podman run --rm --privileged multiarch/qemu-user-static --reset -p yes
|
|
# sudo singularity run docker://multiarch/qemu-user-static --reset -p yes
|
|
|
|
Then, ``wwctl container exec`` will work regardless of the architecture of the container.
|
|
For more information about QEMU, see their `GitHub <https://github.com/multiarch/qemu-user-static>`_
|
|
|
|
To use wwclient on a booted container using a different architecture,
|
|
wwclient must be compiled for the specific architecture. This requires GOLang build
|
|
tools 1.21 or newer. Below is an example for building wwclient for arm64:
|
|
|
|
.. code-block:: console
|
|
|
|
# git clone https://github.com/warewulf/warewulf
|
|
# cd warewulf
|
|
# GOARCH=arm64 PREFIX=/ make wwclient
|
|
# mkdir -p /var/lib/warewulf/overlays/wwclient_arm64/rootfs/warewulf
|
|
# cp wwclient /var/lib/warewulf/overlays/wwclient_arm64/rootfs/warewulf
|
|
|
|
Then, apply the new "wwclient_arm64" system overlay to your arm64 node/profile
|