Because humanity is based on survival. In history, survival would be from being physically fit and being able to catch food and survive to having a visually pleasing appearance. Beauty = Survival. However that is not the only thing that guaranties survival, so does exceptional mental cognition, grand skills in specified activities, and many other things. And when people fail that these tasks, the core of nature find it needed to show that they are not capable of offering survival to others so they should be expelled from the competition of potential mates which offers them greater survival. The general rule is to not do this because it is considered socially unaccepted by most, but some have primitive mind sets and don't even realize this so they punish those who are weaker in fields they excel in.
Plus, I believe that I saw an article a while back that stated that it activates the same area of the brain that activates when you eat chocolate, have sex or succumb to addictions. The pleasure center of the brain dictates what we do more so then most realize in the common day.