I'm writing some software to generate a synthetic load to test a network probe. This code generate UDP packets at a given rate and the probe counts the packets that match a tuple ... same kinda thing as running tcpdump with a filter.

I'm using Intel 82599 NICs, 10GTek 10G transceivers and OM3 fiber.

If connect the load generation box directly to this probe, and send at around 90,000 p/s (roughly 900Mb/s or so) I get 100% of the packets, over hundreds of millions of packets generated.

If I put the CRS305 in the path, with just two of its ports utilized, I will lose sometimes 1, or 2 or 10 packets out of a million, even when only transmitting at around 40,000 p/s. None of the interfaces show any errors or drops on the Mikrotik, same with Linux. I've upgraded the CRS to the latest firmware, 6.48, no change. I've fiddled with flow control, no change.

My expectation is that a truly non-blocking switch would not drop any of these packets, and that if the switch was having throughput issues it would drop *more* than what I'm seeing, not a single packet out of a million. Like I said, I have tested my generation code extensively and when directly connected to this packet probe I get absolute accuracy over hundreds of millions of packets send by the load generator so the problem is not in Linux, nor the generation code.

Before anyone educates me on UDP not being a reliable protocol etc etc, I know that. This is part of a test rig to verify some system integration and as such, I need the sent and received counts to match to pass the test, and they should match if the switch is truly non-blocking and no routers or slow links are in the path.

I'm wondering if anyone has any ideas why this switch would drop a single, or a couple of packets out such a large number.