There's one strange criticism that I hear from Arab Muslims to Arab atheists that really hurts my brain, they mock us for saying sentences and words that are Islamic in nature. Things like "wallahi" (I swear by allah), or "inshallah" (if allah wills), and other stuff like that.

The reason is pretty obvious, those words and sentences are very very common in everyday life, and we don't have a reason to stop saying them or find others to substitute them. We don't believe in an atheist merciful god that will torture us for eternity for saying words that he doesn't like, unlike converts to Islam from a religion like Christianity who stop saying stuff like "jesus christ" because they fear merciful Allah may torture them without mercy for saying such words.