That man made a poor choice and dealing with the consequences of his behavior. You are not at fault for reporting.

My advice is to nip it in the bud with a calm, firm, professional tone, "This is inappropriate. I am not having this conversation." Be direct. Don't leave "hints" or be subtle. He's a grown man and should know better.

You are not responsible for how he reacts to your boundary. No one should ever tell you that "you are making a big deal" or "He was just joking" or "men will be men" or gaslight you in anyway. If anyone makes you uncomfortable, you are within your right to speak up.

Conferences have a code of conduct for attendees and presenters. Speak up to an organizer privately. There should be zero tolerance for any type of harassment.

This applies to anyone. I had another woman made a comment about my large tits in a group in a professional setting. The men were uncomfortable. I just told her, "And why are you focused on my body?" She's like, "I'm just joking." My answer, "No one else is laughing. This isn't appropriate. Change the topic, pls." And then I change the subject.