It may depend on what type of milk it is. Full fat Homogenized milk will have a greasy aftertaste to it because of its fat content. You might consider going to 2% or 1% milk instead. Another factor is that if you're used to drinking full fat milk and then go to skim milk, the skim milk will taste bad to you. People used to drinking skim milk usually think full fat milk tastes bad, and people used to drinking full fat milk think skim milk tastes bad. 2% and 1% fat milk strike a happy medium between the two.