Appeal to the mods

[Sebiale]

This is part of the reason we need block protection on towns--to build walls and have a ground the enemy can't simply dig through, because in reality that's bloody difficult, it's why people prefer to siege castles rather than start hacking at it with pickaxes (or axes in the case of wood).
ETA: Of course, a town should be assailable, though still somewhat difficult. Things like an actual castle should be incredibly difficult. Further, the players probably shouldn't tunnel directly beneath their town, they should mine somewhat away from it or have a special area put aside, otherwise that may open them up to attack.

Once you know the enemy can't just come at you whenever you feel like it, it's actually possible to have a defensible position. Without that defense you're simply too vulnerable to being attacked at any time.

The raiders also have serious consequences for their behavior (or they should anyways)--they're known raiders, which means that in the future any other town or nation should be hesitant about trusting them, and actively forbid trading or otherwise helping them on any level.
This is where metagaming will really become a problem, people will be inclined to forgive people they know IRL, even though the character they play is a dick and shouldn't deserve anything of the sort.