diff --git a/docs/Beowulf-Architecture.png b/docs/Beowulf-Architecture.png new file mode 100644 index 00000000..8c75d919 Binary files /dev/null and b/docs/Beowulf-Architecture.png differ diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..8462357e --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,117 @@ +# Warewulf Cluster Architecture + +## System Architecture +Warewulf's primary design goal is to facilitate the typical architecture of an HPC style cluster. The roots of these +clusters originates from the foundational architecture that has influenced these architectures for the last 20-30 +years, commonly known as the "Beowulf". + +### The Beowulf +The original [Beowulf Cluster](https://en.wikipedia.org/wiki/Beowulf_cluster) was developed in 1996 by Dr. Thomas +Sterling and Dr. Donald Becker at Nasa. The architecture is defined as a group of similar compute worker nodes all +connected together using standard commodity equipment on a private network segment. The control node is dual homed +(has two network interface cards) with one of these network interface cards attached to the upstream network and the +other connected to the same private network which connects the compute worker nodes (as seen in the figure below). + +![Beowulf System Architecture](Beowulf-Architecture.png) + +This architecture is very advantageous for creating a scalable resource for HPC clustering but it is also only a +very simple portrayal of the base foundation. To this we must add storage, scheduling and resource management, +monitoring, interactive systems, and as the system grows, it maybe needed to have groups of nodes, that have +different features, architectures, vintages, memory configuration, GPUs, or interconnects. As the system grows, +the necessity for a scalable control surface also increases which can result in even more complexity. + +### Provisioning +Warewulf is designed to support not only the simplest of clusters, but also the most complicated, largest, and +specialized resources that are demanded today. But in all of these configurations, we go back to this standard +architecture illustrated above. This is because Warewulf is designed to "own" the network broadcast domain due to +how these worker nodes boot. + +As mentioned before, Warewulf is designed first and foremost to be a stateless provisioning subsystem. This means +that the worker nodes maintain no state about their configuration, operating system, or other when they are powered +off. The result of this is when these systems boot, they retain no knowledge on who they are, what they are supposed +to do, or how they got there. For these reasons, these systems will need to boot via +[PXE](https://en.wikipedia.org/wiki/Preboot_Execution_Environment). + +#### PXE +By Default, most servers will have network interface cards that support PXE. In a nutshell, PXE will allow the +network card to be seen by the BIOS as a bootable device. This means that the boot order may need to be configured +in the system's BIOS on the worker nodes to allow it to boot. If there is also another bootable device on these +systems, it might be necessary to set the network interface card to boot first. + +When the system boots via PXE, it will begin a chain reaction of events: + +1. The network card will register an option rom into the BIOS +1. The BIOS will run through all of its functions and finishes with boot devices +1. The boot devices are attempted to be booted in the defined order +1. When it gets to the network boot device, PXE is run from the firmware on the network card +1. PXE will request a Bootp/DHCP on the network +1. If the DHCP response includes a boot file name, it will download this file (iPXE boot image) over TFTP +1. Once iPXE is downloaded, it is loaded by the network card and it will request a configuration from the + server +1. The configuration will tell iPXE what to download and load + +Warewulf manages the entire process once the worker node's network device has begun the PXE process. + +#### Warewulf Server +Warewulf will configure and setup the controller's DHCP and TFTP services, as well as putting all the required +files into place but once the iPXE file has been sent and loaded on the nodes, all network communication from this +point on is being handled by the Warewulf server over HTTP. The order of events from this point on are as follows: + +1. iPXE requests it's configuration and Warewulf generates this on demand from the configured template +1. The default iPXE template tells iPXE to request a kernel, VNFS image, runtime kernel modules, and a system overlay +1. Each of the requested files will be "served" to the node + +#### Post PXE +Once the worker node has received all the required files, the kernel will be booted and the runtime components will be +loaded into the kernel's initial ram file system (initramfs). The order of this is very because each layer is +"layered" on top of the previous. The layers are implemented in the following order: + +1. VNFS/Container image +1. Kernel modules +1. System Overlay +1. Runtime Overlay (repeated at a given interval) + +As part of the provisioning process, the system boots after the System Overlay has been provisioned which means that +the Runtime Overlay occurs after `/sbin/init` has been called and the system. This delineation is important because +it clearly defines what should be in the system Vs. runtime overlay. More on overlays in the next section. + +Before calling `/sbin/init` (or `init` to override in the node configuration file), Warewulf must set up the system. +Again, this process occurs after the System Overlay has been provisioned. This is where the system initializes and +prepares for booting the runtime OS. Depending on the configuration, the system might boot directly in the initramfs +file system as it stands, or it could migrate the root file system to a different mount point (e.g. `tmpfs` or at +some point, hard drives). + +Other things that get done at this stage are setting up and enabling SELinux, IPMI, making any needed changes or +configurations to the file system before booting it, and starting up the `wwclient` which is responsible for loading +the runtime overlay. + +Lastly, the `init` process is executed with PID 1 and thus "boots" the VNFS container. This is where SysVInit, +Upstart, or Systemd takes over. + +#### Warewulf Overlays +As described above, Warewulf uses layers to provision. The first layer is static across any number of nodes, but +each node may require some custom configurations, for example, network. This means there must be a method for +leveraging a base file system which can be shared by many nodes and also be able to configure these file systems +with custom per-node options at a large scale. + +Typically, there are two major times this configuration needs to be done. Pre "boot" and post "boot". In this case, as +described above, we can consider the call to `/sbin/init` the delineation point and the proper way to consider the two +user configurable overlays: system and runtime. + +**System Overlay**: The system overlay is what will be present before `/sbin/init` is called. This gives the +administrator the ability to control the configuration of the booting system itself. For example, network +configuration is necessary to be configured on every node, but each node must have a different network configuration +otherwise the IP addresses will clash and this must be set before Systemd brings up the network device. The Warewulf +system overlay is the right place to configure this. + +**Runtime Overlays**: Some configurations happen after the system boots and continuously at periodic intervals. For +example user and group accounts. You probably don't want to reprovision a node to add a user or change a runtime +configuration, and this is where you will need to use the runtime overlay. + +Both overlays leverage a similar file system template structure. Each overlay (you can create any number of them) +can include text files, directories, links, and templates. Templates allow you to customize any of the content +within an overlay dynamically for each node that will be leveraging that template. + + + +