This phrase irks me for many reasons.

1. Its use suggests that we are dishonest most of the time. Why else would there be a need to clarify the rare occasion of being honest? Are people really so disingenuous to need such a phrase in the vernacular?

2. It is commonly used as a device to deliver comments that would otherwise be rude to say. I’m not really sure why it seems that adding this phrase makes us perceive rude comments as being more polite. If you must express yourself, why not just be blunt? Or why not just withhold your comments?

3. It is often used as a habit in speech e.g. using it after every phrase. In this way it serves no purpose.

4. It is often used to preface good advice. Why not just provide the best advice in the first place? Sadly bad advice is often given when the goal is to make money off of someone’s ignorance.

5. If one MUST use the phrase, it can nearly always be replaced with “honestly,” which is more succinct.