kelvin/warewulf
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #2081 from jsco2t/vagrant-dev-env-v1

Browse Source 
Resolving issues with vagrant dev envionment functionality + updating documentation
This commit is contained in:
Jonathon Anderson
2026-01-08 16:09:13 -07:00
committed by GitHub
parent f2869f4949 1ae132cd7e
commit d0bd5ac63a
12 changed files with 1071 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@@ -32,3 +32,6 @@ userdocs/reference/*
/warewulf-?.?.?/
*.coverprofile
LICENSE_DEPENDENCIES.csv
/vagrant/.vagrant
/vagrant/*.pid

1
CHANGELOG.md
View File

@@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- systemd-networkd overlay with IPv6 support
- `wwctl overlay info` lists the variables used by an overlay template
- Added support for setting mtu and mac address on bonds with NetworkManager #2059
- Added new Vagrant/Libvirt basic test/dev environment #2081, #2003
### Changed

8
userdocs/contributing/development-environment.rst
View File

@@ -95,3 +95,11 @@ required. To use the Dev Container, click the "Open a Remote Window" button on
the bottom left of the editor (`><` icon) and select "Reopen in Container".
This will build the container and open a new VSC window with the container as
the development environment.
Using Vagrant and Libvirt
=========================
Vagrant can be used to quickly spin up a test/development environment for Warewulf.
A `Vagrantfile` is provided in `vagrant` directory of the Warewulf repository.
See the `README.md <https://github.com/warewulf/warewulf/blob/main/vagrant/README.md>`_
for more details.

40
vagrant/.vagrantplugins Normal file
View File

@@ -0,0 +1,40 @@
class InitWarewulf < Vagrant.plugin(2, :command)
def self.synopsis
"Initialize Warewulf Dev Environment"
end
def execute
system "vagrant up"
system "vagrant provision --provision-with provision-vbmc wwctl"
system "vagrant provision --provision-with add-ww-nodes wwctl"
end
end
class CleanupWarewulf < Vagrant.plugin(2, :command)
def self.synopsis
"Cleanup Warewulf Dev Environment"
end
def execute
system "vagrant ssh wwctl -c 'sudo wwctl power off n1' || true"
system "vagrant ssh wwctl -c 'sudo wwctl power off n2' || true"
system "vagrant halt"
system "vagrant destroy -f"
end
end
class InitWarewulfPlugin < Vagrant.plugin(2)
name "InitWarewulf"
command "init-warewulf" do
InitWarewulf
end
end
class CleanupWarewulfPlugin < Vagrant.plugin(2)
name "CleanupWarewulf"
command "cleanup-warewulf" do
CleanupWarewulf
end
end

689
vagrant/README.md Normal file
View File

@@ -0,0 +1,689 @@
# Local Development Environment with Vagrant
This document describes how to get started using `libvirt` and `Vagrant` to facilitate local dev/test environments for Warewulf. There are multiple moving parts involved to getting `libvirt`, `vagrant-libvirt` and `Vagrant` to work in harmony on a Linux host OS. For this reason this document is primarily focused on using Enterprise Linux variants as the host OS. Also note that this dev environment is primarily meant as an example. It's highly likely this example will need to be customized depending on what you are attempting to develop or test.
This document is broken down into the following major topics:
- [Prerequisites](#prerequisites)
- [Overview](#overview)
- [Quick Start](#quick-start) (Environment needs to be configured for `libvirt`/`vagrant-libvirt` first!!!)
- [System Setup](#system-setup-libvirt-and-vagrant-libvirt-installation)
- [Customization](#customization)
- [Development Workflow](#development-workflow)
- [Troubleshooting](#troubleshooting)
## Prerequisites
Before you begin, ensure your system meets these requirements:
### Hardware Requirements
- **CPU**: x86_64 processor with hardware virtualization support (Intel VT-x or AMD-V)
- **RAM**: Minimum 8 GB (16 GB recommended)
- wwctl VM: 2 GB
- wwnode1: 768 MB (for Alpine) or 2+ GB (for EL/SLES)
- wwnode2: 768 MB (for Alpine) or 2+ GB (for EL/SLES)
- **Disk Space**: Minimum 20 GB free space in `/var/lib/libvirt/images`
### Software Requirements
- **Operating System**: Linux host OS (Enterprise Linux 9/10 or openSUSE Leap 16 preferred)
- **System Setup**: Complete the [System Setup](#system-setup-libvirt-and-vagrant-libvirt-installation) section before running Quick Start commands
### Working Directory
All commands in this guide should be run from this `vagrant/` directory within the Warewulf repository unless otherwise specified.
### Files in This Directory
This directory contains all the necessary files for the Vagrant environment:
- `Vagrantfile`: VM configuration defining wwctl and compute nodes
- `.vagrantplugins`: Custom Vagrant commands (`init-warewulf`, `cleanup-warewulf`)
- `export-libvirt-sock.sh`: SSH tunnel script for libvirt socket forwarding
- `alpine-bootable.def`: Apptainer definition for Alpine Linux container
- `alpine-boot.sh`: Script to build and deploy Alpine to compute nodes
- `init-scripts/`:
- `init-wwctl.sh`: Warewulf server initialization script
- `add-ww-nodes.sh`: Node registration script
- `start-vbmcd.sh`: Virtual BMC startup script
## Overview
This Vagrant environment provides a self-contained Warewulf cluster for development and testing. It consists of three VMs orchestrated via libvirt and installs Warewulf version 4.6.4 by default (configurable via `WW_VERSION` environment variable):
- **wwctl** (Rocky Linux 9): Control node running Warewulf services (DHCP, TFTP, NFS) on network 10.100.100.0/24. Includes virtualbmc for IPMI emulation and apptainer for container support. Configuration files are provisioned to the home directory of the vagrant user.
- **wwnode1/wwnode2**: Network-boot-only compute nodes (no management network) configured to PXE boot from wwctl.
Key architectural components:
1. **SSH Reverse Tunnel**: An SSH tunnel (`export-libvirt-sock.sh`) exposes the host's libvirt socket (`/var/run/libvirt/libvirt-sock`) into the wwctl VM at `/var/tmp/libvirt.sock`, enabling the guest to manage sibling VMs.
2. **Virtual BMC**: Two vbmc instances run on wwctl, providing IPMI interfaces (ports 6231/6232) for power control of wwnode1/wwnode2 via the tunneled libvirt connection.
3. **Custom Vagrant Commands**: The `.vagrantplugins` file defines `init-warewulf` (orchestrates VM creation, vbmc setup, and node registration) and `cleanup-warewulf` (powers off nodes and destroys environment).
4. **Automated Provisioning**: The `init-wwctl.sh` script grows the root partition, installs Warewulf (version controlled via `WW_VERSION` env var), configures services, and sets up the vbmc systemd service. Nodes are registered via `add-ww-nodes.sh` with MAC addresses matching the Vagrantfile definitions.
The result is a functional Warewulf cluster where `wwctl power on/off n1/n2` commands control the compute nodes through IPMI, and the nodes boot their operating system images via network provisioning.
## Quick Start
**Note**: All commands in this section should be run from the `vagrant/` directory.
### Environment Setup and Teardown
The recommended way of setting up and tearing down the dev environment is by using the custom Vagrant commands: `init-warewulf` and `cleanup-warewulf`.
- `init-warewulf`: Command provisions the Vagrant environment with three virtual machines (a Warewulf Server and two Compute Nodes). **Note:** neither compute node has an OS deployed to them at this point.
- `cleanup-warewulf`: Command completely tears down the Warewulf development environment.
### Basic Workflow
1. Setup the development environment:
```shell
vagrant init-warewulf
```
This command will take 5-10 minutes to complete. It performs the following:
- Downloads the Rocky Linux 9 base image (first time only)
- Creates and provisions the wwctl VM with Warewulf services
- Creates wwnode1 and wwnode2 VMs
- Sets up Virtual BMC for IPMI emulation
- Registers the compute nodes with Warewulf
Once this command completes executing you will have three nodes running in vagrant:
- `wwctl`: The Warewulf server node (fully configured and running)
- `n1`: The first compute node (powered off, no OS deployed)
- `n2`: The second compute node (powered off, no OS deployed)
2. Deploy Alpine Linux to the compute nodes:
```shell
vagrant ssh wwctl -c "sudo ./alpine-boot.sh"
```
This script will take 2-3 minutes to complete. It performs the following:
- Creates an Alpine container image using apptainer from `alpine-bootable.def`
- Imports the resulting image into Warewulf and builds it
- Creates an Alpine specific network overlay
- Creates an _"alpine"_ Warewulf profile that uses this new network overlay
- Assigns the alpine image and profile to nodes `n1` and `n2`
- Power-cycles both nodes (Warewulf provisioning takes over and deploys the OS via network boot)
You should see the nodes power on and boot into Alpine Linux via PXE.
3. Verify the compute nodes are running:
```shell
vagrant ssh wwctl -c "sudo wwctl node list"
vagrant ssh wwctl -c "sudo wwctl node status n1,n2"
```
Expected output from `node list`:
- Both n1 and n2 should appear in the list with their configurations
Expected output from `node status`:
- Shows IPMI power state, network boot status, and system information
- Nodes should show as "POWERED ON" if boot was successful
4. Cleanup the environment:
```shell
vagrant cleanup-warewulf
```
### Power-cycling Compute Nodes
As a precondition, shell into the `wwctl` vagrant VM:
```
vagrant ssh wwctl
```
To then power-cycle the compute nodes you would do the following:
```shell
sudo wwctl power off n1
sudo wwctl power off n2
sudo wwctl power on n1
sudo wwctl power on n2
```
## System Setup: `libvirt`, `vagrant-libvirt`, and `Vagrant` Installation
**Note**: Vagrant bundles its own embedded Ruby environment in Vagrant. The Vagrant package installer may not always recognize system dependencies. Instructions for working around this are included for `Enterprise Linux 9 and 10` below as well for `openSUSE Leap 16`. There are other options for running `vagrant-libvirt`. For more information please see the [vagrant-libvirt](https://vagrant-libvirt.github.io/vagrant-libvirt/) site.
### `libvirt` Install
#### EL 9/10 Instructions
```shell
# Verify KVM support (should show kvm_intel or kvm_amd)
lsmod | grep kvm
# Install virtualization packages
sudo dnf groupinstall -y "Virtualization Host"
sudo dnf install -y qemu-kvm libvirt virt-install virt-top libguestfs-tools
# Enable and start libvirtd
sudo systemctl enable --now libvirtd
# Add your user to the libvirt group (critical for non-root usage)
sudo usermod -aG libvirt $(whoami)
# Verify libvirt works before proceeding
sudo virsh list --all
```
#### openSUSE Leap 16 Instructions
```shell
# Verify KVM support (should show kvm_intel or kvm_amd)
lsmod | grep kvm
# Install virtualization packages
# Note: openSUSE uses patterns instead of groups
sudo zypper install -t pattern kvm_server kvm_tools
# Install additional virtualization tools
# (many are included in the patterns, but explicit install ensures they're present)
sudo zypper install -y qemu-kvm libvirt virt-install virt-top guestfs-tools
# Enable and start libvirtd
sudo systemctl enable --now libvirtd
# Add your user to the libvirt group (critical for non-root usage)
sudo usermod -aG libvirt $(whoami)
# Verify libvirt works before proceeding
sudo virsh list --all
```
**IMPORTANT**: After completing the above, log out and log back into the system (or run `newgrp libvirt`) to refresh your group membership. This is required for non-root access to libvirt.
### `libvirt` / `virsh` Configuration
```shell
# Check and start default network
sudo virsh net-list --all
sudo virsh net-start default 2>/dev/null || true
sudo virsh net-autostart default
# Create default storage pool if missing
sudo virsh pool-define-as default dir --target /var/lib/libvirt/images
sudo virsh pool-build default
sudo virsh pool-start default
sudo virsh pool-autostart default
```
To run read-only `virsh` commands without `sudo`, set this environment variable:
```shell
export LIBVIRT_DEFAULT_URI="qemu:///system"
```
To make this permanent, add the above line to your `~/.bashrc` or `~/.zshrc`.
### Vagrant Install
#### EL 9/10 Instructions
```shell
# Install repository management tools
sudo dnf install -y dnf-plugins-core
# Add HashiCorp repository
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
# Install Vagrant
sudo dnf install -y vagrant
# Verify installation
vagrant --version
```
#### openSUSE Leap 16 Instructions
```shell
# Set desired Vagrant version (check https://releases.hashicorp.com/vagrant/ for latest)
VAGRANT_VERSION="2.4.9"
# Download the RPM
wget https://releases.hashicorp.com/vagrant/${VAGRANT_VERSION}/vagrant-${VAGRANT_VERSION}-1.x86_64.rpm
# Install with zypper
sudo zypper install ./vagrant-${VAGRANT_VERSION}-1.x86_64.rpm
# Verify installation
vagrant --version
```
**Note**: Version 2.4.9 is known to work. Check the [Vagrant releases page](https://releases.hashicorp.com/vagrant/) for newer versions.
### `vagrant-libvirt` Configuration
**Precondition:** Install `libvirt-devel` to support building of the `vagrant-libvirt` plugin:
#### EL 9/10 Instructions
```shell
# Enable CRB repository (required for libvirt-devel)
sudo dnf config-manager --set-enabled crb
# Install development tools and all required libraries
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y \
libvirt-devel \
libxml2-devel \
libxslt-devel \
ruby-devel \
gcc \
gcc-c++ \
make \
cmake \
zlib-devel \
pkgconf-pkg-config \
byacc \
wget \
rpm-build
```
Now, install `vagrant-libvirt`:
```shell
# Set CONFIGURE_ARGS to help locate libvirt
export CONFIGURE_ARGS="--with-libvirt-include=/usr/include/libvirt --with-libvirt-lib=/usr/lib64"
# Install the plugin
vagrant plugin install vagrant-libvirt
```
#### openSUSE Leap 16 Instructions
```shell
# Minimal essential set for vagrant-libvirt
sudo zypper install -y \
libvirt-devel \
libxml2-devel \
libxslt-devel \
ruby-devel \
gcc \
gcc-c++ \
make
```
Now, install `vagrant-libvirt`:
_Note: Vagrant's embedded libraries are conflicting with the system shell on openSUSE Leap 16. When the build runs pkg-config, Vagrant sets LD_LIBRARY_PATH to its embedded libs, and its bundled libreadline.so.8 breaks /bin/sh with a symbol lookup error. To work around this we temporarily have to rename the `libreadline` library in Vagrant so that the system lib path is used._
```shell
# Temporarily rename the problematic library
sudo mv /opt/vagrant/embedded/lib/libreadline.so.8 /opt/vagrant/embedded/lib/libreadline.so.8.bak
# Set CONFIGURE_ARGS to help locate libvirt
export CONFIGURE_ARGS="--with-libvirt-include=/usr/include/libvirt --with-libvirt-lib=/usr/lib64"
# Install the plugin
vagrant plugin install vagrant-libvirt
# Restore the readline library
sudo mv /opt/vagrant/embedded/lib/libreadline.so.8.bak /opt/vagrant/embedded/lib/libreadline.so.8
```
### Verifying setup
First, let's verify Vagrant reports the plugin is now present:
```shell
vagrant plugin list
```
You should see `vagrant-libvirt` in the output.
Next - run a simple test to verify that Vagrant can use libvirt to deploy a VM:
1. Create a temporary test directory (outside of the warewulf vagrant directory):
```shell
mkdir -p ~/vagrant-test
cd ~/vagrant-test
```
2. Create a `Vagrantfile` with the following contents:
```ruby
Vagrant.configure("2") do |config|
config.vm.box = "rockylinux/9"
config.vm.box_version = "5.0.0"
config.vm.provider :libvirt do |libvirt|
libvirt.cpus = 2
libvirt.memory = 2048
libvirt.nic_model_type = "virtio"
libvirt.machine_virtual_size = 32
end
end
```
3. Start the VM (this will download the Rocky Linux 9 box if not already present):
```shell
vagrant up --provider=libvirt
```
You should see output indicating successful VM creation and provisioning.
4. Verify SSH access:
```shell
vagrant ssh -c "cat /etc/os-release"
```
You should see Rocky Linux 9 version information.
5. Clean up the test:
```shell
vagrant destroy -f
cd ~
rm -rf ~/vagrant-test
```
If all steps completed successfully, your system is ready to use the Warewulf Vagrant environment!
## Customization
### Warewulf Version
By default, the environment installs Warewulf version 4.6.4. To use a different version, set the `WW_VERSION` environment variable before running `vagrant init-warewulf`:
```shell
export WW_VERSION="4.6.5"
vagrant init-warewulf
```
Check the [Warewulf releases page](https://github.com/warewulf/warewulf/releases) for available versions.
### Compute Node Memory
The default configuration allocates 768 MB per compute node, which is sufficient for Alpine Linux. If you plan to test with Enterprise Linux (Rocky, RHEL, AlmaLinux) or SLES, you'll need more memory.
Edit the `Vagrantfile` and modify the `domain.memory` value for wwnode1 and wwnode2:
```ruby
# For EL or SLES distributions
config.vm.define :wwnode1 do |node1|
# ... other config ...
node1.vm.provider :libvirt do |domain|
domain.memory = 2048 # Increase from 768 to 2048 or higher
# ... other settings ...
end
end
```
### Network Configuration
The default network configuration uses 10.100.100.0/24:
- wwctl: 10.100.100.254
- DHCP range: 10.100.100.2 - 10.100.100.9
To customize the network, edit:
1. `Vagrantfile`: Update the IP address for the wwctl private network
2. `init-scripts/init-wwctl.sh`: Modify the `/etc/warewulf/warewulf.conf` network settings
### Adding More Compute Nodes
To add additional compute nodes, edit the `Vagrantfile` and add a new VM definition:
```ruby
config.vm.define :wwnode3 do |node3|
node3.vm.network :private_network, :libvirt__network_name => 'netboot', :libvirt__mac => '006e6f646533'
node3.vm.synced_folder '.', '/vagrant', disabled: true
node3.vm.provider :libvirt do |domain|
domain.memory = 768
domain.cpus = 1
domain.mgmt_attach = false
boot_network = {'network' => 'netboot'}
domain.boot boot_network
end
end
```
**Note**: The MAC address pattern `006e6f646533` translates to "node3" in hex. Adjust the last digits accordingly for additional nodes.
You'll also need to:
1. Update `init-scripts/add-ww-nodes.sh` to register the new node
2. Update `init-scripts/init-wwctl.sh` to add a vbmc instance for the new node
## Development Workflow
### Testing Warewulf Code Changes
If you're developing Warewulf itself and want to test your changes:
#### Method 1: Using a Custom RPM
1. Build your modified Warewulf RPM locally
2. Copy the RPM to the vagrant directory
3. Modify `init-scripts/init-wwctl.sh` to install your local RPM instead of downloading from GitHub:
```shell
# Replace this line:
# dnf install -y https://github.com/warewulf/warewulf/releases/download/v${ww_version}/warewulf-${ww_version}-1.el9.$(arch).rpm
# With:
dnf install -y /vagrant/your-custom-warewulf.rpm
```
4. Re-provision the environment:
```shell
vagrant destroy -f
vagrant init-warewulf
```
#### Method 2: Live Development with Synced Folders
For faster iteration during development, you can enable synced folders to mount your Warewulf source code:
1. Edit `Vagrantfile` and enable the synced folder for wwctl:
```ruby
wwctl.vm.synced_folder '/path/to/your/warewulf/source', '/home/vagrant/warewulf-dev'
```
2. Install Warewulf from your local source inside the VM:
```shell
vagrant ssh wwctl
cd ~/warewulf-dev
# Build and install Warewulf from source
```
This allows you to edit code on your host and immediately test changes in the VM without rebuilding RPMs.
### Accessing Warewulf Logs
Common log locations on the wwctl node:
```shell
vagrant ssh wwctl -c "sudo journalctl -u warewulfd -f" # Warewulf service logs
vagrant ssh wwctl -c "sudo journalctl -u dhcpd -f" # DHCP logs
vagrant ssh wwctl -c "sudo journalctl -u tftp -f" # TFTP logs
vagrant ssh wwctl -c "sudo tail -f /var/log/messages" # System logs
```
### Testing Different Container Images
The example uses Alpine Linux, but you can create and test other distributions:
1. Create an Apptainer definition file (e.g., `rocky-bootable.def`)
2. Build and import the container:
```shell
vagrant ssh wwctl
sudo apptainer build rocky-bootable.sif rocky-bootable.def
sudo wwctl container import rocky-bootable.sif rocky
sudo wwctl container build rocky
```
3. Assign to nodes:
```shell
sudo wwctl node set n1,n2 --container rocky
sudo wwctl power cycle n1,n2
```
### Snapshot and Restore
Libvirt supports VM snapshots, which can be useful for testing:
```shell
# Create a snapshot of the wwctl node
virsh snapshot-create-as vagrant_wwctl wwctl-snapshot1
# List snapshots
virsh snapshot-list vagrant_wwctl
# Revert to snapshot
virsh snapshot-revert vagrant_wwctl wwctl-snapshot1
# Delete snapshot
virsh snapshot-delete vagrant_wwctl wwctl-snapshot1
```
## Troubleshooting
### Common Issues
#### "Cannot connect to libvirt" or Permission Denied
**Problem**: You get permission errors when running vagrant commands.
**Solution**:
1. Verify you're in the `libvirt` group: `groups | grep libvirt`
2. If not, ensure you logged out and back in after adding yourself to the group
3. Alternatively, run `newgrp libvirt` to refresh group membership without logging out
4. Verify libvirtd is running: `sudo systemctl status libvirtd`
#### Vagrant Init Fails During Box Download
**Problem**: The Rocky Linux box download fails or times out.
**Solution**:
1. Check your internet connection
2. Manually download the box: `vagrant box add rockylinux/9 --box-version 5.0.0 --provider libvirt`
3. Try again: `vagrant init-warewulf`
#### Compute Nodes Won't Boot
**Problem**: After running `alpine-boot.sh`, nodes don't power on or fail to boot.
**Solution**:
1. Check IPMI connectivity:
```shell
vagrant ssh wwctl -c "sudo ipmitool -I lanplus -H 10.100.100.254 -p 6231 -U admin -P password power status"
```
2. Verify vbmc is running:
```shell
vagrant ssh wwctl -c "sudo systemctl status vbmcd"
vagrant ssh wwctl -c "sudo vbmc list"
```
3. Check the libvirt SSH tunnel:
```shell
ps aux | grep libvirt.sock
```
4. Check DHCP/TFTP logs:
```shell
vagrant ssh wwctl -c "sudo journalctl -u dhcpd -f"
vagrant ssh wwctl -c "sudo journalctl -u tftp -f"
```
#### "Network netboot not found" Error
**Problem**: Vagrant fails to create VMs with network errors.
**Solution**:
1. Verify the netboot network exists:
```shell
virsh net-list --all
```
2. If missing, vagrant should create it automatically. Try:
```shell
vagrant destroy -f
vagrant init-warewulf
```
#### Out of Disk Space
**Problem**: VM provisioning fails due to disk space issues.
**Solution**:
1. Check available space: `df -h /var/lib/libvirt/images`
2. Clean up old vagrant boxes: `vagrant box prune`
3. Remove unused libvirt images:
```shell
virsh vol-list default
virsh vol-delete <volume-name> default
```
#### vagrant-libvirt Plugin Won't Install
**Problem**: `vagrant plugin install vagrant-libvirt` fails during compilation.
**Solution**:
1. Ensure all development libraries are installed (see System Setup section)
2. Check that `CONFIGURE_ARGS` is set correctly
3. For openSUSE, verify the libreadline workaround was applied
4. Check build errors for missing libraries and install them
#### Compute Nodes Stuck at "Powered Off"
**Problem**: `wwctl node status` shows nodes as powered off even after power on commands.
**Solution**:
1. Manually power on via IPMI:
```shell
vagrant ssh wwctl -c "sudo ipmitool -I lanplus -H 10.100.100.254 -p 6231 -U admin -P password power on"
```
2. Check vbmc logs:
```shell
vagrant ssh wwctl -c "sudo journalctl -u vbmcd -f"
```
3. Verify libvirt can see the nodes:
```shell
vagrant ssh wwctl -c "sudo LIBVIRT_DEFAULT_URI='qemu:///system?socket=/var/tmp/libvirt.sock' virsh list --all"
```
#### Failed to Cleanup Environment
**Problem**: `vagrant cleanup-warewulf` hangs or fails.
**Solution**:
1. Manually power off nodes via libvirt:
```shell
virsh destroy vagrant_wwnode1
virsh destroy vagrant_wwnode2
virsh destroy vagrant_wwctl
```
2. Force destroy:
```shell
vagrant destroy -f
```
3. Clean up SSH tunnel:
```shell
pkill -f "libvirt.sock"
rm .export-libvirt-sock.pid
```
### Additional Resources
- **Warewulf Documentation**: https://warewulf.org/docs/v4.6.x/troubleshooting/troubleshooting.html
- **vagrant-libvirt Documentation**: https://vagrant-libvirt.github.io/vagrant-libvirt/
- **Libvirt Documentation**: https://libvirt.org/docs.html
- **Warewulf GitHub Issues**: https://github.com/warewulf/warewulf/issues

58
vagrant/Vagrantfile vendored Normal file
View File

@@ -0,0 +1,58 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
ENV['VAGRANT_NO_PARALLEL'] = 'yes'
Vagrant.configure("2") do |config|
ww_version = ENV['WW_VERSION'] || "4.6.4" # set this env-var to adjust which version of WW will be installed
config.vm.define :wwctl do |wwctl|
wwctl.vm.box = "rockylinux/9"
wwctl.vm.box_version = "5.0.0" # see: https://git.resf.org/infrastructure/meta/issues/138
wwctl.vm.network :private_network, :ip => "10.100.100.254", :libvirt__dhcp_enabled => false, :libvirt__network_name => 'netboot'
wwctl.vm.synced_folder '.', '/vagrant', disabled: true
wwctl.trigger.after :up, :reload do |trigger|
trigger.run = {path: "export-libvirt-sock.sh"}
end
wwctl.vm.provider :libvirt do |domain|
domain.memory = 2048
domain.cpus = 2
domain.machine_virtual_size = 32
end
wwctl.vm.provision "init-wwctl", type: "shell", path: "init-scripts/init-wwctl.sh", args: [ww_version]
wwctl.vm.provision "file", source: "alpine-bootable.def", destination: "alpine-bootable.def"
wwctl.vm.provision "file", source: "alpine-boot.sh", destination: "alpine-boot.sh"
wwctl.vm.provision "provision-vbmc", type: "shell", run: "never", path: "init-scripts/start-vbmcd.sh"
wwctl.vm.provision "add-ww-nodes", type: "shell", run: "never", path: "init-scripts/add-ww-nodes.sh"
end
# Ram here is specific to the size of the ww node that's starting up. For alpine 768mb is fine - for EL or SLES
# you will need to allocate a higher value for `domain.memory`.
config.vm.define :wwnode1 do |node1|
node1.vm.network :private_network, :libvirt__network_name => 'netboot', :libvirt__mac => '006e6f646531'
node1.vm.synced_folder '.', '/vagrant', disabled: true
node1.vm.provider :libvirt do |domain|
domain.memory = 768
domain.cpus = 1
domain.mgmt_attach = false
boot_network = {'network' => 'netboot'}
domain.boot boot_network
end
end
config.vm.define :wwnode2 do |node2|
node2.vm.network :private_network, :libvirt__network_name => 'netboot', :libvirt__mac => '006e6f646532'
node2.vm.synced_folder '.', '/vagrant', disabled: true
node2.vm.provider :libvirt do |domain|
domain.memory = 768
domain.cpus = 1
domain.mgmt_attach = false
boot_network = {'network' => 'netboot'}
domain.boot boot_network
end
end
end

47
vagrant/alpine-boot.sh Executable file
View File

@@ -0,0 +1,47 @@
#!/bin/sh
if [ -d /tmp/alpine-rootfs ]; then
rm -rf /tmp/alpine-rootfs
fi
apptainer build -s /tmp/alpine-rootfs alpine-bootable.def
wwctl image import --force /tmp/alpine-rootfs alpine
wwctl image build alpine
wwctl power off n1
wwctl power off n2
if ! wwctl overlay list | grep "alpine-net" >/dev/null; then
wwctl overlay create alpine-net
wwctl overlay mkdir alpine-net /etc/network
mkdir -p etc/network
cat > etc/network/interfaces.ww <<EOF
{{- range \$devname, \$netdev := .ThisNode.NetDevs }}
# This file is autogenerated by warewulf
{{- if \$netdev.OnBoot.BoolDefaultTrue }}
auto {{ \$netdev.Device }}
{{- end }}
iface {{ \$netdev.Device }} inet static
address {{ \$netdev.Ipaddr }}
netmask {{ \$netdev.Netmask }}
{{ if \$netdev.Gateway }}gateway {{ \$netdev.Gateway }}{{ end }}
{{ if \$netdev.MTU }}mtu {{ \$netdev.MTU }}{{ end }}
{{ end -}}
EOF
wwctl overlay import alpine-net etc/network/interfaces.ww
fi
if ! wwctl profile list -j | jq -e 'to_entries[] | select(.key=="alpine")' >/dev/null; then
wwctl profile add --profile=default alpine
wwctl profile set -y alpine --system-overlays=$(wwctl profile list -j | jq -r 'to_entries[] | select(.key=="default") | .value."system overlay" | join(",")' | sed 's/debian.interfaces/alpine-net/')
fi
wwctl node set -y --image=alpine --profile=alpine n1
wwctl node set -y --image=alpine --profile=alpine n2
wwctl overlay build n1
wwctl overlay build n2
wwctl power on n1
wwctl power on n2

15
vagrant/alpine-bootable.def Normal file
View File

@@ -0,0 +1,15 @@
Bootstrap: docker
From: alpine:3.22
%post
apk add linux-lts linux-firmware-none openrc mdevd-openrc alpine-conf
setup-devd udev || true
passwd -d root
rc-update add networking boot
rc-update add modules
setup-sshd openssh
ln -s /bin/sh /usr/bin/sh
cat > /etc/ssh/sshd_config.d/empty-password.conf <<EOF
PermitRootLogin yes
PermitEmptyPasswords yes
EOF

14
vagrant/export-libvirt-sock.sh Executable file
View File

@@ -0,0 +1,14 @@
#!/bin/sh
pidfile=".export-libvirt-sock.pid"
pid=$(cat $pidfile 2>/dev/null || true)
if [ "$pid" != "" ]; then
if grep "/var/tmp/libvirt.sock:" /proc/$pid/cmdline >/dev/null 2>&1; then
exit 0
fi
fi
nohup vagrant ssh wwctl -- -o ServerAliveInterval=5 -o ServerAliveCountMax=1 -f -R /var/tmp/libvirt.sock:/var/run/libvirt/libvirt-sock -N >/dev/null 2>&1 &
echo $! > .export-libvirt-sock.pid

4
vagrant/init-scripts/add-ww-nodes.sh Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/sh
wwctl node add -G 10.100.100.1 -H 00:6e:6f:64:65:31 --ipaddr 10.100.100.10 --ipmiaddr 10.100.100.254 --ipmipass password --ipmiport 6231 --ipmiuser admin --ipmiinterface lanplus --netdev eth0 --netmask 255.255.255.0 --nettagadd="DNS1=10.100.100.1" n1
wwctl node add -G 10.100.100.1 -H 00:6e:6f:64:65:32 --ipaddr 10.100.100.11 --ipmiaddr 10.100.100.254 --ipmipass password --ipmiport 6232 --ipmiuser admin --ipmiinterface lanplus --netdev eth0 --netmask 255.255.255.0 --nettagadd="DNS1=10.100.100.1" n2

189
vagrant/init-scripts/init-wwctl.sh Executable file
View File

@@ -0,0 +1,189 @@
#!/bin/sh
#
# Resize root partition (if necessary)
#
dnf install -y cloud-utils-growpart util-linux bc
echo ""
echo "================================================================================"
echo "[Resizing Root Partition]"
echo ""
# Find the device that is mounted as root (/)
ROOT_DEVICE=$(findmnt -n -o SOURCE /)
echo "Detected root device: $ROOT_DEVICE"
# Resolve any symlinks or device-mapper names to the actual partition
ROOT_PARTITION=$(readlink -f "$ROOT_DEVICE")
echo "Detected root partition: $ROOT_PARTITION"
# Extract the parent disk and partition number from the partition path
PARENT_DISK="/dev/$(lsblk -no PKNAME "$ROOT_PARTITION")"
PARTITION_SUFFIX="${ROOT_PARTITION#$PARENT_DISK}"
PARTITION_NUMBER="${PARTITION_SUFFIX#p}"
echo "Parent disk: $PARENT_DISK"
echo "Partition number: $PARTITION_NUMBER"
# Check if there's actually room to grow
# Note: We use tail -n +2 to skip the first line, which is the disk itself.
# Lines 2 onward are the actual partitions.
DISK_SIZE_BYTES=$(lsblk -b -dn -o SIZE "$PARENT_DISK")
PARTITIONS_SIZE_BYTES=$(lsblk -b -ln -o SIZE "$PARENT_DISK" | tail -n +2 | paste -sd+ | bc)
UNALLOCATED_BYTES=$((DISK_SIZE_BYTES - PARTITIONS_SIZE_BYTES))
THRESHOLD_BYTES=$((100 * 1024 * 1024))
echo "Disk size: $((DISK_SIZE_BYTES / 1024 / 1024)) MiB"
echo "Allocated to partitions: $((PARTITIONS_SIZE_BYTES / 1024 / 1024)) MiB"
echo "Unallocated space: $((UNALLOCATED_BYTES / 1024 / 1024)) MiB"
if [ "$UNALLOCATED_BYTES" -gt "$THRESHOLD_BYTES" ]; then
echo "Growing partition $PARENT_DISK partition $PARTITION_NUMBER..."
growpart "$PARENT_DISK" "$PARTITION_NUMBER"
FS_TYPE=$(lsblk -no FSTYPE "$ROOT_PARTITION")
echo "Detected filesystem type: $FS_TYPE"
case "$FS_TYPE" in
xfs)
xfs_growfs /
;;
ext4|ext3|ext2)
resize2fs "$ROOT_PARTITION"
;;
*)
echo "Warning: Unknown filesystem type '$FS_TYPE'. Cannot auto-resize."
echo "Partition was grown, but you may need to manually resize the filesystem."
;;
esac
echo "Root partition successfully grown!"
df -h /
else
echo "No significant unallocated space available. Nothing to do."
fi
echo "================================================================================"
echo ""
echo "StreamLocalBindUnlink yes" > /etc/ssh/sshd_config.d/60-forward-cleanup.conf
systemctl reload sshd.service
dnf install -y libvirt-client python3-pip python3-libvirt ipmitool epel-release
pip3 install virtualbmc
dnf install -y apptainer
ww_version=$1
cat > /usr/local/bin/setup-wwnode <<EOF
#!/bin/sh
node=\$1
port=\$2
if [ ! -e /root/.vbmc/\$node ]; then
/usr/local/bin/vbmc add --username admin --password password --port \$port --address 10.100.100.254 --libvirt-uri qemu:///system?socket=/var/tmp/libvirt.sock \$node
fi
/usr/local/bin/vbmc start \$node
EOF
chmod 755 /usr/local/bin/setup-wwnode
dnf install -y https://github.com/warewulf/warewulf/releases/download/v${ww_version}/warewulf-${ww_version}-1.el9.$(arch).rpm
cat > /etc/warewulf/warewulf.conf <<EOF
ipaddr: 10.100.100.254
netmask: 255.255.255.0
network: 10.100.100.0
warewulf:
port: 9873
secure: false
update interval: 60
autobuild overlays: true
host overlay: true
grubboot: false
api:
enabled: false
allowed subnets:
- 127.0.0.0/8
- ::1/128
dhcp:
enabled: true
template: default
range start: 10.100.100.2
range end: 10.100.100.9
systemd name: dhcpd
tftp:
enabled: true
tftproot: /var/lib/tftpboot
systemd name: tftp
ipxe:
00:0B: arm64-efi/snponly.efi
"00:00": undionly.kpxe
"00:07": ipxe-snponly-x86_64.efi
"00:09": ipxe-snponly-x86_64.efi
nfs:
enabled: true
export paths:
- path: /home
export options: rw,sync
- path: /opt
export options: ro,sync,no_root_squash
systemd name: nfs-server
ssh:
key types:
- ed25519
- ecdsa
- rsa
- dsa
image mounts:
- source: /etc/resolv.conf
dest: /etc/resolv.conf
readonly: true
paths:
bindir: /usr/bin
sysconfdir: /etc
localstatedir: /var/lib
cachedir: /var/cache
ipxesource: /usr/share/ipxe
srvdir: /var/lib
firewallddir: /usr/lib/firewalld/services
systemddir: /usr/lib/systemd/system
datadir: /usr/share
wwoverlaydir: /var/lib/warewulf/overlays
wwchrootdir: /var/lib/warewulf/chroots
wwprovisiondir: /var/lib/warewulf/provision
wwclientdir: /warewulf
EOF
systemctl enable --now warewulfd
wwctl configure --all
systemctl restart warewulfd
cat > /etc/systemd/system/vbmcd.service <<EOF
[Install]
WantedBy = multi-user.target
[Service]
ExecReload = /bin/kill -HUP $MAINPID
ExecStart = /usr/local/bin/vbmcd --foreground
Group = root
Restart = on-failure
RestartSec = 2
TimeoutSec = 120
Type = simple
User = root
ExecStartPost = /usr/local/bin/setup-wwnode vagrant_wwnode1 6231
ExecStartPost = /usr/local/bin/setup-wwnode vagrant_wwnode2 6232
[Unit]
After = syslog.target
After = network.target
Description = vbmc service
EOF
systemctl daemon-reload
# shut down the firewall - not needed for this test environment:
sudo systemctl disable --now firewalld

3
vagrant/init-scripts/start-vbmcd.sh Executable file
View File

@@ -0,0 +1,3 @@
#!/bin/sh
systemctl enable --now vbmcd.service