Hats are for outdoors as a protection from the weather. They're not needed indoors so they should be removed. And indoors or outdoors, you should NEVER eat with a hat on. Drives me crazy when I see that.
Most of the time it is not, as far as I know. But certain places and events may have a dress code that prohibits the wearing of hats indoors probably because it lacks class more than just being rude. It is also a military tradition to take headgear off in buildings.