While its not impossible it was a bug, many people think normal game mechanics are bugs so there could be more to it.
For one, how old was the door? If it was a new door it would have been easy to break, same goes for any walls that you may have just built.
also, you said he destroyed loot – there’s no real way to destroy loot in this game, do you mean he picked it up and took it?
Is it possible that the door he broke was touching a wall he built?
Not to blame you, but there are other possibilities that should be ruled out. Poorly built bases don’t withstand raiders well, bases with nature tiles, road tiles, etc. are easy for anyone to get into with a bit of planning.