Greetings!

I'm currently using Proxmox and Proxmox makes SR-IOV Networking a pain, much moreso than it needs to be. I'm looking at maybe switching to XCP-NG, or even just refining my Proxmox configuration. I have pretty basic requirements. I need Proxmox style HA, and I need offline migration.

I totally get why (It's painfully obvious) that you can't live migrate a VM with SR-IOV hardware. It would require migrating physical hardware state to the new VM. And While I'm sure a dedicated appliance vendor with 1 NIC model and support from the NIC manufacturer could pull it off, it's obviously outside the realm of a general purpose Type 1 Hypervisor OS.

Proxmox HA doesn't require online migration. It explicitly waits for a host to die and be fenced, and then brings up a new copy of the VM elsewhere. This is all I really require, is that I can either power off a host and the VMs will spin up again elsewhere, or that I can shut it down, migrate it, and bring it back up again, in approximately the time it takes to transfer the VM metadata (I have entirely shared storage). No Live migration, but Proxmox makes this difficult, as I have to add a systemd unit to each system and configure the MAC address of a VF, and then detach a VF from a VM, and migrate it and attach a new VF on the new machine, double checking a bunch of documentation and references along the way to make sure I'm using the *right* VF. A simple solution would be adding UI to say "Choose the first free VF from a NIC matching this regex, set this MAC and that VLAN, and attach it to the VM. Fail if VFs are exhausted", but Proxmox is focused in a different direction.

So I'm reading through the documentation for VFs in XCP-NG. It *looks* at a glance like VFs work this way, but I'm not familiar enough with XCP-NG to be sure. https://xcp-ng.org/docs/networking.html#network-troubleshooting

Given my goals, is XCP-NG likely to be able to provide that offline migration with VFs ? If not XCP-NG, is ESXI likely a better choice? If Neither, then how do I go about simplifying migration and HA in Proxmox with VFs attached? It can't really be this difficult. This was *designed* to simplify HPC in large DC environments with RDMA offload, TOE, FCoE, and other high performance technologies, making them easy to deploy at the VM scale. I feel like I'm missing a huge piece of the SR-IOV puzzle.