Because the advantage of evolving with the genetics to remove the remnants of the mammary gland would not be greater than the advantage of "being lazy" and not removing them. They do no harm, and so there is no evolutionary pressure to remove them. Same with the belly button. It gives no advantage, but it also does not hinder an organism's reproduction. Basically, the reason is because it does no harm, so there was no reason for evolution to leave men without them.