You are reinforcing this mistake by telling yourself this. You are doing just fine with the symbolic representation of phonemes (um... we call them letters), and math and music are just other languages with different letters that your brain is perfectly capable of understanding. Like any skill however, it takes practice.
It's not hard. For example: 6+x=10. '+' and '-' are opposites and since '+' is being used in the equation, you find the answer by going '10-6=x'. That's a really simple one, but if you can understand that, you should be able to develop the skill just fine.