To offer an alternative perspective, we encountered similar at a point in our journey. I was blessed not to have the performance issues a number of the couples we got together with did and am pretty well endowed myself which led to me having a great time while my wife was hitting a certain frustration point.

There were good experiences sprinkled in there, but for the majority of PIV play I was definitely getting the upper hand, which is not to say we didn't have friends who could perform and a couple that were better endowed than myself and I enjoyed these experiences and enjoyed watching her pleased...

Then we had one experience with a single male with a reputation for being huge, who I knew she'd played with in separate rooms at a party--we invited him over where it was just the three of us and the dude's dick was like a forearm... He was a legit 11 inches and watching my wife cum instantly when he inserted and make sounds I hadn't heard from her before was very off putting... I was used to being 'Mr. Big' I don't think it was jealousy that I felt--perhaps a 're-alignment of ego' but it rubbed me weird for a couple of days and I didn't react as well as I should have.

Of course I got over it and we've played with him several times since and now have a couple of other freakishly endowed friends, and those feelings subsided to the point I enjoy watching it, but I can totally see your husband having weird and conflicting feelings at hearing your reaction "having the time of your life" like that for the first time and it throwing off his own head.

I'd say give him the benefit of the doubt, let him process and try again.