There are plenty of simple illustrations of this sort of situation that can easily be implemented on larger scales. Take the farmer/soldier example that you provided, for instance. Let's say that in order to develop things (and learn how to operate that which has already been developed), some sort of time investment must be made. So, the farmer would spend his or her time invested in the farm. If a thief kills the farmer and takes the farmer's food, the thief will starve after being unable to continue the farm's operation (out of a lack of expertise in farming and farm equipment).

Let's add in another level of complexity. Let's say that there are a lot of people who become farmers in order to reap the benefits of the production that farming can bring. And because they're not idiots, they stick together for mutual defense. Now, the thief can probably get away with stealing from the farm's peripheries. But the moment that they get near the farmers themselves (to kill and rob them, let's say), all of the farmers can band together for mutual defense and will either drive the thief out or kill them.

Let's add in another level of complexity. Let's say that nobody's special. Neither farmers nor anybody else can work a full-fledged farm on their own, and none of the thieves are Rambo. The only ones who can succeed at whatever they try to do are the ones who do it with other people. This brings us to organizational systems.

Looking at this through a more complex level: Groups of farmers will beat groups of thieves. Farmers banding together for mutual defense face the tragedy of the commons: if a thief attacks, each farmer is incentivized to not risk their neck in defending another's land from the thief while hoping that all the other farmers would choose to risk their necks instead. This is a bad situation, but the situation facing the thieves is even worse. Although farmers face the tragedy of the commons, the thieves face the prisoner’s dilemma (the game theory, not as prisoners). In fact, the number of prisoner’s dilemma games that the thieves face is n! where n=thief population. If they move in a coordinated assault, each thief will be tempted to steal from the others and hope that none of them steal back. And take into account that stolen objects can be stolen back, so the only way to resolve the dilemma is for thieves to kill the other thieves. So while the farmers only have to overcome the tragedy of the commons, the thieves have to overcome a hell of a lot of prisoners’ dilemmas.

And going beyond this, farming communities can produce a small but reliable surplus that can be used for specialization. In the early stages, this can be used to provide for guards, who are like thieves but are more specialized because they do not have to hunt for food to supplement whatever the thieves get from stealing. And as the farm settlement expands, it can provide enough of a surplus to provide for several craftsmen or artisans, and civilization develops from there.

And pop goes the weasel, that takes you from thieves and farmers to the rest of history. I may have made some mistakes, and I appreciate your feedback.

EDIT: I might have messed up the prisoner's dilemma bit, but I can't imagine how it wouldn't get violent. They're either doing the prisoner's dilemma or some weird inverted tragedy of the commons, but either way it gets real ugly real fast unless the thieves were somehow super organized.

EDIT 2: Never mind, it's definitely a prisoner's dilemma. A weird, immediate change of answer for either party style prisoner's dilemma. It's the free-for-all of the game theory world, I guess.