This is actually something brought up in his background. His rival as a child was a girl, and as they grew up, she grew curvier and curvier, making it harder and harder to keep ahead with the boys of the dojo. Breasts are a hindrance to a fighter, and woman warriors should dread growing them if they want to stay on equal footing with men in combat. Poor lady Zoro.
