Constant Robber Baron Attacks

Now that I don't have my Beginner's Protection any longer, I am getting constant attacks from robber baron castles. I am assuming that this is at least the third time I've been attacked, even when I'm not doing anything. Robber Baron attacks are weak and easy to defeat, but they still always manage to kill a few of my soldiers, which becomes annoying. I do not think robber baron castles even have agents, so they can never espionage you (at least at my level), meaning you can't catch an enemy agent to help prepare for their attack. I guess that's why you always have to constantly train more soldiers. This is just  a heads-up note for those who still have Beginner's Protection and don't have to worry about this yet.