If you have a fortress the ultimate layout of which is the one I drawn here as an example, and you have tribe doors in the yellow tiles, and stone walls in the grey tiles, then I think that the stone walls are the weak point for any number of attackers.
I don’t know the relative value of scales and stone; that value would help understand when you should stop building outer layers and start reinforcing the inner ones — supposing you have enough space.
A point in favor of reinforcing rather than layering is that you will repair a larger portion of your defenses per repair kit use.
A point in favor of having a huge external layer is privacy.