Oh oh oh I know this!
I work in the refurb division of an ewaste recycling company[0]. To prepare a machine for sale, the drive needs to be wiped, and (optionally) an OS loaded. Wiping happens in WipeOS[1], which loads when you PXE boot on the internal company network. To install an OS, I have a separate network on my desk that will load iVentoy[2] when PXE booted, where I can further boot from ISOs I have on my server, but I almost always install Linux Mint. With those 2 things, I can largely do my job without fumbling with and losing USB drives.
I have 2 16 port switches on my desk, with over a dozen ethernet cables plugged into each. The yellow cables will PXE boot WipeOS, and the black ones PXE boot iVentoy.
Ah I did something similar at the university I worked at as a student. Everything was already set to network boot as the first step, so I set up a PXE server that loaded up DBAN. When we needed to wipe a lab before decommissioning, we'd flip their network to the PXE DBAN network, tell them all to reboot, and decom them in the morning.
Saved us a bunch of hours we then used to play Minecraft haha
I've seen this done in some settings as well; the 'wipe and install the system' VLAN and the 'normal behaviour' VLAN. When you want to reinstall a server you tell it to reboot and then swap the VLAN; once the installation is done you swap it back.
Alternately, you can have your DHCP server be aware of all of your systems and know which ones need to be reinstalled; then just configure every server to network boot by default with a short timeout. If the DHCP server thinks your system needs to be erased then it serves a boot/wipe/reinstall image; otherwise, it doesn't and the system continues to boot normally.
Now that I think about it, I think our admin system for the DHCP server was how we handled it instead of VLANs. Also helped with automated installation of Windows on desktops, bootstrapping servers, etc.
I'm guessing you use WipeOS to more easily handle securely erasing disks. Could you have iVentoy host WipeOS to simplify the setup?
Yes, but then you need to select the proper menu option at boot time. Sometimes just moving the hardware stack one to the left and swapping the cables is quicker.
Yes, WipeOS does secure erasing, and can do some mild diagnostic tests (something like MemTest86, but not as extensive). WipeOS can format a bootable USB drive to boot WipeOS if PXE doesn't work for whatever reason (BIOS doesn't support it, bug with NIC, etc). It should be possible to create an ISO from said USB drive, but I'm the only one who uses iVentoy. I haven't made that ISO because I don't mind swapping the ethernet cable during the ~10 seconds it takes to reboot.
I've set up PXE booting at two previous companies for very different use cases.
The first was to automate server deployment; we ran bare metal servers, and even though we had managed hosting in our data centre the installation, configuration, and deployment of a server could potentially take days since it was just me doing it and I had other things to do.
So one day I set to work. I installed an Ubuntu server the same way I always did and then captured the debconf configuration to turn into a preseed file. I set up the disk partitioning, etc., and configured the OS to boot from DHCP. Then I configured the DHCP server with MAC addresses for every server we got and an associated IP address so that a given physical server would always get the same IP address.
Then I set up an internal apt repository; that's where I put custom packages, backports I had to recompile, third-party packages (e.g. perconadb) and so on.
Lastly, I set up salt (the management orchestration tool, like puppet or chef or ansible) with a nice simple (detailed) configuration.
The machines would be configured to boot via PXE. They'd load the kernel and initrd, which contained the preseed file that answered all of the installation/configuration questions. Lastly it ran the post-install shell script which started salt and ran the initial configuration, much of which was based on hostname. This would turn the current DHCP-provided IP address into a static networking configuration so that the server wasn't reliant on DHCP anymore; it would ensure that SSH keys were installed, and that the right services were enabled or disabled, install some packages based on the hostname (which represented the role, e.g. db02.blah.blah got percona installed). I also had some custom data sources (whatever you would call them) so that I could install the right RAID controller software based on which PCI devices were present; after all that, it would reboot. Once it rebooted from the local disk, salt would pick back up again and do the rest of the configuration (now that it wasn't running from a chroot and had all the required systemd services running). What used to take me several days to do for two servers turned into something one of our co-ops could do in an hour.
Second was another company that wanted to standardize the version of Linux its developers were running. Again, I set up an Ubuntu installer and configured it to boot iPXE and then fetch the kernel and the root image via HTTPS. The Ubuntu installer at that point was a Snap, and the default 'source' was a squashfs file that it unpacked to the new root filesystem before proceeding with package installation. I set up some scripts and configurations to take the default squashfs filesystem, unpack it, install new packages via apt in a chroot, and then repack it again. This let me do things like ensure Firefox, Thunderbird, and Chrome were installed and configured not from snaps; update to the latest packages; make sure Gnome was installed, etc. A lot of that was stuff the installer would do, of course, but given we were on gigabit ethernet it was significantly faster to download a 2 GB squashfs file than to download a 512M squashfs file and then download new or updated packages. One again what used to start with "Here's a USB, I think it has the latest Ubuntu on it" and take most of a day turned into "Do a one-off boot from the network via UEFI, choose a hostname, username, and password, and then just wait for twenty minutes while you get a coffee or meet your coworkers". I even found a "bug" (misbehaviour) in the installer where it would mount the squashfs and then rsync the files, which seemed significantly slower because the kernel was only using one thread for decompressing; using `unsquashfs` could use all cores and was dramatically faster, so I got to patch that (which I'm not sure ever made it into the installer anyway).
The one thing I couldn't make work was the OEM installation, where you put everything down onto the system unattended then put the user through the Ubuntu OOBE process. That would have made it far easier to pre-provision systems for users ahead of time; I did replace the default Plymouth splash screen logo with our company logo though, which was pretty cool.
I also set up network booting of macOS at another job, but that's sort of a very different process because it has all its own tooling, etc. for managing and Apple ended up moving from custom deployment images to static images and MDM for post-install configuration.
TL;DR network booting is pretty great actually; it's a very niche use case but if you're clever you can get a lot done. There's also lots of options for booting into a bootloader that can then present other options, allowing you to choose to netboot Ubuntu Desktop, Ubuntu Server, Windows, RHEL, Gentoo, a rescue image, or anything else you want.
> The one thing I couldn't make work was the OEM installation, where you put everything down onto the system unattended then put the user through the Ubuntu OOBE process.
Did you try chain booting into iPXE and using SYSLINUX?
I used just nginx try, where I could place a pressed for a known provisioning event, otherwise providing various live and utility images if the MAC address file didn’t exist for one off or emergency repair.
I could even serve up windows instances.
That is also very useful because occasionally you run into PXE firmware that is crippled, it may not apply now, but only having a tiny iPXE image on tftp helps with speed and security.
I would bet almost all vendors just use iPXE anyway, and at least you use to be able to replace the firmware on intel cards with it.
The fun thing about learning to boot from PXE, is that you have to learn it every time you onboard a new type of hardware... or a new VM hypervisor... or new NIC firmware... or new BIOS firmware.
God help you if you actually want to install an operating system.
PXE is such a vital capability for working with on-prem servers. But it's ten different things which all have to play nicely together. Every time I build a PXE system I feel like I'm reinventing the universe in my tiny subnet.
I've not found this at all -- PXE "just works" on legacy boot or UEFI for me. I've used it for years to install hosts via Foreman (https://theforeman.org/), as well as for personal stuff on my home network, and it's so much better than getting people to use USB sticks or whatever else!
Agreed, PXE seems ideal for provisioning things, but it's just too hard to use, especially when you're not on a network you fully control.
I just want to start the computer, and have it download an immutable OS image from somewhere I decide (and supply a checksum for, etc). I don't want to set up TFTP or any of this other stuff. It feels like I should be able to just specify an IP (let's say) a checksum (maybe supply that information to the NIC directly somehow), and be off to the races after a reboot.
replace the PXE stack with an OS installer written in UEFI. This bootload can be installed through a guest running on the host in the EFI partition, or possibly through PXE or direct UEFI http load.
this allows you intermediate the boot process without coordinating with the administrative owner of the DHCP server, and is actually less janky than PXE
Not to sound incredibly lazy (because I am), but is there anything off the shelf that does this? Anyone doing something similar would also be great -- I saw some UEFI-in-Rust projects recently so maybe it's not too hard to hack through myself.
That said, my original need for this was on Hetzner, and what I did instead was actually completely automate their reset setup (thankfully they have an API to that), so I have a solution, but I would much rather if I could prebake and load images much easier.
I think these days Hetzner has much more UEFI support across it's dedicated server fleet.
Side note: Every time I see tinkerbell and other relatively new PXE boot projecs and it's set of tools I feel relief, then dig in, then feel dread again around iPXE.
Side note 2: I really want to do two things:
1. Load OSes from the network easily
2. Run the OS in RAM (ECC)
I feel like it would take my server management experience to the next level -- I've spent an inordinate amount of time messing with Hetzner USB add-ons and Alpine to try to get things to work, but it wasn't reliable (it worked, but wasn't reliable).
If you're familiar with the Linux boot process, you may be aware that the system often does run from RAM for a period of time, before mounting a filesystem from a block device and calling pivot_root to transfer the rootfs over.
If you want to run the OS from RAM, the absolute simplest way to do it is to simply never transfer. Building a custom initramfs has never been easier. There are tools explicitly geared towards making an initramfs, like Dracut, but also a huge ecosystem of tools for building container images which could almost certainly be scripted to spit out a cpio without a lot of trouble. You can even use something like Buildroot.
As far as loading it from the network easily, I would also look into Unified Kernel Images (UKIs). They combine the EFI stub, the kernel, and the initramfs into one single file. You should be able to load that directly from the PXE firmware.
Yeah I'm famliar with initramfs-es (well at least enough to be dangerous)! I've run Alpine from ram before but it just wasn't stable and I ended up going back to a more traditional and perfectly fine setup.
> If you want to run the OS from RAM, the absolute simplest way to do it is to simply never transfer. Building a custom initramfs has never been easier. There are tools explicitly geared towards making an initramfs, like Dracut, but also a huge ecosystem of tools for building container images which could almost certainly be scripted to spit out a cpio without a lot of trouble. You can even use something like Buildroot.
This certainly seems like a lot of responsibility to take on, I'm just surprised there isn't already a distro that is very good at doing everything it does (with modern ease of use) with the caveat of the OS disk being in RAM. Technically there are (there's a whole list on Wikipedia), but last I tried it just didn't work well for me.
These days there are more build-your-own distro options like CoreOS, Flatcar, linuxkit that absolutely make things even easier but they don't have that final bit of being runnable from RAM.
I question whether it's worth it to maintain my own UEFI/initramfs/boot setup rather than just building (or pulling off the shelf) an immutable OS image and using the reset + wipe automation to flash the disks once in a blue-moon when reconfiguring/scaling machines up/down.
> As far as loading it from the network easily, I would also look into Unified Kernel Images (UKIs). They combine the EFI stub, the kernel, and the initramfs into one single file. You should be able to load that directly from the PXE firmware.
This is certainly interesting -- haven't looked into these much, but this would certainly be useful in the PXE firmware (or custom UEFI). But my problem here is that iPXE is dependent on support at the provider level (and Hetzner does not make this available to end users, though they use it internally obviously when you reset), so the other commenter's suggestion:
> replace the PXE stack with an OS installer written in UEFI.
Would not work with this approach. The suggestion is interesting though, custom installer that pulls down a UKI sounds viable from my particular armchair.
Tails works like this, and has gained a lot of traction in the past few years — although one may argue that running from RAM is only indirectly responsible for its popularity.
I think this idea appeals to many people, also concerning remanence: keeping your system and user data separate to the point that you could virtually mount your /home on any given UNIX host, with the added bonus that if the host is not compatible with your setup, you can always reboot it on your USB stick, run a live ISO on RAM, and retrieve a decent work environment.
True PXE doesn't require coordination with the DHCP server.
https://en.wikipedia.org/wiki/Preboot_Execution_Environment
The network client boot stack sends a DHCPDISCOVER as a broadcast. Any machine can be listening on UDP 67 (bootps) for this. The real DHCP server responds with the DHCPOFFER containing the IP address the client should use. Around the same time, the PXE server responds with its own DHCPOFFER that does not issue an address, but does contain the values for the requested DHCP options.
The client basically keeps broadcasting DHCPDISCOVER until it gets both, then it does the unicast DHCPREQUEST and wait for unicast DHCPACK with the normal DHCP server.
Now, that said, I've only ever seen this work with commercial PXE servers like Microsoft RIS. To my knowledge, ISC DHCPD is unable to send a DHCPOFFER with options but no address. But my knowledge is at least a decade out of date.
At home I just set the options on the main DHCP server like every other hobbyist does, but this isn't true PXE, this is just plain old DHCP+TFTP remote boot.
Let's say you do have such a server that sends DHCPOFFER with the options and no IP address. If it's on its own machine, then it can listen on port 67, same as the real DHCP server on another machine. But, if it's on the same machine as the DHCP server, it has to listen on port 4011. In this case the client behaves a little differently. For this to work, the DHCP server must send as part of the DHCPOFFER an unsolicited option 60 to indicate that the client should go ahead and accept the IP then send a second unicast DHCPDISCOVER to port 4011 and await a DHCPOFFER from that port. Option 60 is only needed, and can only be used, if the independent PXE server is running on the same host as the DHCP server.
So there's basically 3 scenarios: * Hobbyist: just configure the booting options on the real DHCP server * Real PXE, separate machine: Both real DHCP and PXE listen for broadcast DHCPDISCOVER and respond with complementary DHCPOFFER. Real DHCP server has no knowledge whatsoever about booting. * Real PXE, same machine: Real DHCP server responds with unsolicited option 60 no matter what. This is the extend of its knowledge of booting. Separate PXE server runs on port 4011 instead of 67, and everything is unicast.
There may finally be hobbyist projects that support this model, but when I last did this stuff, there were not. Learning how RIS worked was a revelation for me, and it really made me wonder why the hobbyist community of the time seemed hellbent on not doing PXE correctly, which annoyingly requires control over the options set by the real DHCP server, and often makes it impossible to do fun stuff like use different boot files for different clients.
I’m confused, are you talking about getting PXE enabled in the hardware, or customizing something about your PXE software for the new hardware?
There's a lot of nonsense at every level. Especially when dealing with heterogenous infrastructure.
Some NICs support http. Some NICs support tftp. Some NICs have enough memory for a big iPXE, other NICs don't. Some BMC systems make next-boot-to-lan easy, but not all.
We almost always use iPXE in order to normalize our pxe environment before OS kickstart. There's a lot to it and quite a lot of little things that can go wrong. Oh, and every bit of it becomes critical infra.
Ok, that makes more sense. I'm used to iPXE, and I guess that quick bootstrap from PXE->iPXE bypasses a lot of the nonstandard weirdness.
All of 'em.
we need to go /stalinmode/ on the whole bootup and initialization industry subsector. it should be required by law for that stuff to be open source and documented.
"but muh competitive advantage??"
its literally a for loop that reads sectors from disk/network into memory and jumps to the start address.
if a local build of the (vendor provided source code) firmware doesn't match the checksum of the build thats flashed on the actual mobo, you get sent to a cobalt mine.
Boot by committee (UEFI) doesn't seem much better than boot by fiat (BIOS). For everything nice it gives you, you lose something nice that BIOS gave you ... or you have something nice that you lose when you exit boot services. Or there's an extension for something nice that isn't usable on mainstream hardware.
UEFI gives you nicer video modes, but not a text mode after boot services.
UEFI has an extension for booting images from the network, but afaik, it's impossible to use, and there's no reasonable way to boot from a disk image; working UEFI network boot has to pull pieces out of the filesytem and present them seperately; as opposed to MEMDISK which makes the image available as a BIOS disk and the image is labeled so that one the OS is loaded, the image can be used without BIOS hooks. If this is possible on UEFI generally, it isn't widely distributed knowledge. Something that will work on any UEFI system that makes it to iPXE, subject to changes to the OS in the image (which is reasonable... MEMDISK needs changes too, unless the OS runs all disk I/O through BIOS APIs)
It's been a long while since I did anything with UEFI, but my recollection is that the standard data structures are reasonably well documented, especially when they are meant to be part of booting an OS
I imagine that whatever UEFI extension implements loading a disk image over the network probably also implements some way of knowing where the sectors are in RAM so that the OS bootloader can choose to hand off access to the memory disk to the OS.
Is such support implemented in any bootloaders? I have no idea. My guess is probably not because people would rather just have the bootloader use the available services to download the disk image itself.
You're getting downvotes for being hyperbolic about it, but boot integrity is really both a consumer safety and a national security issue.
Yeah in order to automate, you’ve gotta know something about what you’re automating. PXE is not different.
PXE is awesome, especially if you combine it with systemd's UKI mechanism and its EFI stub. You can load a single file via TFTP or HTTP(S) and boot into a read-only (or ramdisk-only) full Linux system. Most off the shelf distributions can be made to work in this way, with a small bit of effort. A very usable Debian system is a few hundred MB.
You can extend this with secure boot (using your own keys) to sign the entire UKI file, so your firmware will authenticate the full "disk" image that it boots into.
I've used PXE (not even iPXE, just DHCP/TFTP without HTTP) mainly in environments where a LAN client-server game would need to be launched on many systems at once. Nothing quite like rolling out a hand-tailored distro for a single game to 16 computers and seeing them all boot and load straight into the game, one after the other, entirely unattended, from one broadcast boot-over-Ethernet trigger.
I think at one point we were even using distcc to use the clients to speed up rebuilds while iterating on the game. I should revisit that with iPXE and icecream.
TFTP is crazy slow, even with RFC 7740 (buffering), but the payloads are usually small so few people care.
Thankfully modern BIOSes tend to implement HTTP boot option, where you can point to any HTTP or HTTPS URL (as long as the URL ends with ".efi", which is a pretty dumb limitation if you ask me).
You can also do things like boot with PXE (Legacy or UEFI PXE boot) to get a small image like iPXE, and then have iPXE do the http boot part. This means that you have an extra shim but you can pull larger images than TFTP is any good for.
TFTP is also UDP and I don't think it is pipelined, so it's all req->ack->req->ack, so any additional latency hits it hard too.
They let you boot off HTTPS? That explains why corp IT pushed out a Dell BIOS vulnerability update today relating to OpenSSL in my BIOS.
Yup! You can point your BIOS at a .efi and it’ll Just Boot It. We’ve even got Wi-Fi support in some of these as well for a full wireless deploy…
https://www.dell.com/support/manuals/en-us/bios-connect/http...
Having done lots of network booting over the years, here are a few of my lessons learned:
PXE is a big improvement over the boot EPROMs that we needed to install on our NICs back in the day. Those would get an address via DHCP and then TFTP the boot image, and boot it.
I've had some trouble with PXE boot that's been caused by STP. If your PXE boot server has, or is behind a bridge with STP turned on, it can prevent the client from booting. I think this has something to do with the STP "learning state", but turning off STP on the bridge can solve the problem, as long as you're sure that you will not be creating any network loops on the affected interfaces.
There's also a new "https boot", which is supposed to be a PXE replacement, but TLS certs have time validity windows, and some clients may not have an RTC, or might have a dead CMOS battery, and those might not boot if the date is wrong.
> There's also a new "https boot", which is supposed to be a PXE replacement, but TLS certs have time validity windows, and some clients may not have an RTC, or might have a dead CMOS battery, and those might not boot if the date is wrong.
I think the lack of entropy right after boot can also be a problem for the RNG. But, maybe that has been solved in more modern hardware.
You don't need to turn off STP, usually it's enough to set the forward delay to a very small value ("port fast" in cisco commands). If there is a loop, the port will usually still detect it, you at the most get a handful of multiplied packets.
And all the "http boot" firmware I've seen either always ignores certificate errors or doesn't do TLS anyways.
Lots of fun with this and lots of possibilities.
Had great experience using PXE to boot HPC farms, mounting the OS from a NAS and using only a local disk in the machine for tmp and other writable locations. I am not sure how 'diskless' linux works these days on rocky flavours but was solid in centos 5 through 7.
I've had issues using root-on-nfs now that dnf uses a sqlite database. However, root-on-iscsi has been working really well for me.
At first I was PXE booting a kernel and custom initramfs over TFTP, and having the initramfs initiate the iscsi session. More recently I have been PXE booting iPXE, which then has support iscsi. iPXE initiates the iscsi session, reads the kernel and initramfs off the "disk", then once the kernel and initramfs have loaded, the initramfs uses some magic called iBFT to "rediscover" the session that the firmware initiated.
The nice part about that second approach is I don't need to set up any hooks to transfer a new kernel and initramfs to my server whenever they're updated on the machine.
Ive always used the feature(?) allowing you to mount parts of the fs read-write persistent on the local disk which might help the sqlite issue, though not sure if thats a read/write issue or something else.
Have never used scsi for this. i suspect it's faster?
> Ive always used the feature(?) allowing you to mount parts of the fs read-write persistent on the local disk which might help the sqlite issue, though not sure if thats a read/write issue or something else.
The entire nfs based rootfs is mounted rw at this point, so it's not simply a matter of it being a read-only filesystem. NFS has some weird file locking behavior that makes it behave noticeably different from a local filesystem for certain functions. Some sources seem to claim NFSv4 has better support for file locking, but I was unable to get root-on-nfs working reliably under any permutation of NFS parameters I attempted.
Loading the NFS rootfs wasn't a problem though, so I figure a distro that was a bit more intentional about not making nfs-unfriendly software part of the core would be able to do it. Although I also tried setting up Arch in this way and had similar issues.
> Have never used scsi for this. i suspect it's faster?
I'm not really sure whether it's faster. A lot of online sources say "iscsi is faster because block device" and it's not immediately obvious to me how that would make it faster.
I would say that in my experience iSCSI has had fewer compatibility issues. Applications seem to be happier with a high latency block device and a traditional filesystem, than doing file operations over the network.
I’m glad a lot of server stuff has redfish. But something better still needs to be there for non-server stuff for sure. Raspberry pi style bootloaders would be amazing, ones we could configure to use a certain image before powering on for first boot would be even more amazinger.
I figured out how to PXE boot 20,000 PS5 APU blades (BC-250) during covid when I couldn't even get to the actual hardware. Great fun.
[flagged]