We are all the same sex, or asexual, during the early stages of development. It's not until later that we diverge into either male or female, so we share all the same basic "pieces", but they're just arranged differently.
The choice of gender in the body plan of developing humans is more of an end-game decision than an initial choice. We all start out with a common body plan- hearts lungs, hands, feet, etc, and most of our development is well established before finishing touches like gender are established. Boy/Girl is not a coin flip of our genes, it is a complex summation of many competing influences (like testosterone and estrogen balance) and the scales commonly tip in one direction or another, but there are also individuals who fall less distinctly across that gender boundary and represent the entire spectrum of these influences.
The 'default' body plan is female (in mammals anyway- birds did it differently) and only become male at a relative late developmental stage if the right signals are received.
Vestigial breasts in males are one of the clearest signs of this characteristic default embryonic template that in males just never gets the order to grow into breasts.
When your in the womb sex is determined by X and Y chromosomes you start out in Y chromosome (female) and when the X turns on (male) you still have the original nipples when the Y chromosome was dominant
We all start out as females, in the womb. If a Y chromosome is present and activated, the female develops into a male, if only the x is present, the embryo stays female. The human male sperm is responsible for what sex the child will be.