please be gentle with me, I still feel emotionally raw

I made a big mistake yesterday & got myself all worked up & anxious, tried to smoke to alleviate some of the anxiety but ended up just getting more paranoid, especially when I reached out to my usual supports. Usually they remind me I'm being crazy/irrational or remind me that my relationship is amazing & my boyfriend (J) would never hurt me but instead they validated my fears & it made me feel even more insecure & anxious. This led to several panic attacks & eventually my first time making my boyfriend actually mad at me. We are ENM, but exes & mutual friends are off limits for sexual relations.

Saturday I asked J if he wanted to come to my son's soccer game Sunday, he said he would try but he had made plans with his ex fiance (C) to get ice cream since she was in town. We've talked many times about my insecurities/jealousy of this friendship. My issues are: (1) she broke it off with him because she decided she didn't want a family & he did, now he is resigned to not having a family & doesn't want kids or marriage. They decided they were better off as friends. (2) He was on/off with his ex wife while they dated. On one of the off times he sought her out for a friendly ear & "one thing led to another, due to this ex wife banned communication with her. (3) As soon as he was divorced he struck up the friendship again.

Anyway, Sunday came along & he was unable to make soccer. He texted me as he was getting ready, but as soon as they were together he went radio silent for 5 hours. They were supposed to go out for an ice cream. So I finally sent a text that I was going to take my meds & go to bed because I had been feeling anxious. He called about 15 mins later but I had the beginning of a panic attack & had to get off the phone. After I calmed down I tried calling but he didn't get back to me for 45 minutes. By then I'd worked myself up again & my rational brain told him that I was feeling irrationally angry about how the day played out & that we should talk when I wasn't feeling so emotional.

Unfortunately my irrational brain was just angry when he said that was fine & he'd talk to me tomorrow & from there it blew up a bit because I felt like I couldn't just cry myself to sleep & leave this unresolved. My friends were validating my fears & I felt he was acting nonchalant about the whole thing. I felt his response time was incredibly slow for the level of emotion I was feeling & ended up making him mad by being accusatory. I was triggered from the lack of response all day because of relationship trauma I am working through with my therapist. An ex used to hang out with his exes because they were "just friends" and now I feel I've punished him for someone else's behavior. I also worked myself up into several panic attacks when I've done so good not having any for a long time & when the anxiety ended, I felt severely depressed because of my behavior.

I also have this niggling thought about how he wanted to see her for support also because of a new health diagnosis & he said he spent a lot of their time together sad about it. I rationally know I can't be everything for him, but it also hurts a little that he turned to her for support & a shoulder to literally cry on but not me the day before.

Tldr I'm insecure about this relationship & I'm not sure how to manage that. I also plan on apologizing for my behavior last night in person later. Any and all advice is welcome.