Is the sizing on the Bartlett all over the place, or is the variety in personal taste just that varied?

I’m 5’10”, 180 lbs, so I ordered a medium and small Bartlett off of Prime Wardrobe to give em a test run. I’m normally a medium, but even with a sweatshirt on underneath, I still feel like I’m swimming in it. A small could use an extra inch of length at the waist and sleeve and same could be said for the torso width, but of the two it’s the better fit (even with a sweatshirt underneath). I’ve seen people 5’7” and 165 lbs, people the same size as me, and people 6’ and 200 lbs all say the medium is perfect, so it just seems all over the place.