Welcome to the U.S., buddy! Women are allowed to do practically anything, but if men do some if the stuff women do it's 'gross' or 'perverted'. Like wearing booty shorts. Those don't even look good on girls, but when guys wear them they get suspended from school, etc.
I think technically it is a social impoliteness if either guys or gals wear their hats indoors, but it also depends on the surrounding scenario, it's a worse impoliteness in some instances and generally overlooked in others.