Should I be Scared of Him or am I Overreacting?

My STBXH has been verbally and emotionally abusive for some time. This escalated to mild physical abuse (throwing objects across the room, breaking into locked doors to get to me and yell at me more, punching holes in walls, driving recklessly with me in the car, etc.). He also shoved me once. He sexually assaulted me once.

After quite a while of this, I eventually asked for a divorce in January of this year. Unfortunately, he still hasn't signed our (agreed upon) separation agreement, so as to not risk marital abandonment, I can't leave until he does. My attorney said I don't have concrete evidence to support that my life is in danger because it's just he said she said.

But things have escalated. He has shoved me again and also pushed me against a table hurting my finger. (Bear in mind, I'm trying to be as quiet and peaceful as I can and I'm not instigating this, he's just angry all the time)

Twice now this month, I've woken up to him standing over me in the dark just staring at me while I sleep (I'm sleeping on the couch in the living room)

One of my friends says I should leave and I'm not safe, but his parents and my parents all say that these things aren't that bad (I didn't tell my parents about the SA, just my MIL and she didn't believe me).

Should I be scared of him or no? Should I just leave and risk marital abandonment or no? Bear in mind, we have two young children, so it's hard for me to just leave.

I'm just not sure...