Find attached a brief diagram of my network to help the understanding.

https://pasteboard.co/fNZ2ZpfwSbGA.png

So I'm testing 10gbe NICs on my desktop, and pretty much everything is going well. I'm mostly using iperf3 to test, and am considering anything over 7 gigabit successful, because load and variability, etc. Most systems give a solid 9 end to end. Well, I'm testing between izanami, my desktop, and ingot, my TrueNAS core NAS, virtualized on Midori, my core VM server. I only get around 3Gbit. So I assume something's wrong with my desktop, the virtualization stack, or the VM. Fair enough, right? Well, to figure out which, I run tests between all the VMs and hosts shown in the diagram in almost every direction. From ao to ingot, izanami to ao, izanami to midori, and even izanami to test01, a VM I spun up on another VM. even izanami to inazuma. and every connection gives me a solid fast connection. But this one connection pair, izanami to ingot (Yes, I know, a lot of i names), no matter what I do, tops out at roughly 3Gbit. So it's not the link from desktop to switch. It's not the link from Midori to switch, and it's not a problem with the ovs bridge on midori, and it's not a problem with the virtual device on ingot. I've tested all of those parts. They all work when used in any other pair of devices. I've tried adding more -P(arallel connections), or running in -R(everse mode), and none of that changes anything. This pair of hosts is slow. Nothing I can figure out to do to make them faster.

does anybody have any tips or advice?

A few more details:

Midori and ao are R620 running Proxmox with Intel X520DA2 rNDC cards installed.

Ingot is TrueNAS Core

Inazuma is Debian 11

Izanami (Desktop) is Windows 10 with a Mellanox Connectx-3 EN Pro card

Switch is an icx66100

There is absolutely no routing between these devices, they're all on the same VLAN, and the links are mostly idle relative to the 10Gbe throughput I'm testing.

Edit (Update/Resolution):

So after getting almost NO responses, and putting in all the effort to make the nice diagram to help understand everything, I came back a few days later to continue diagnosing. So a little more detail: Izanami originally had a 10g Mellanox ConnectX-2 card for testing while I waited for my ConnectX-3s for the whole house of desktops, and it was working fine. I switched it out to a Mellanox ConnectX-3 when the card came in, but there was one important setting I forgot to apply. MTU 9600. I set this, and now everything just works. I *suspect* the other end was sending MTU 9600 packets and then having to fragment down to 1500 in the switch every packet in the stream, and that was causing some slowdown, because something somewhere wasn't getting the 'must fragment' signal and/or wasn't acting on it properly and yeah. So the solution was to set MTU=9600 on the desktop, and now it works.