Most people don't think about their beliefs, they just go along with the herd. It is a mental shortcut that works more often than not -- if most people are doing it, it is probably right.
This is especially powerful when it comes to religion because religion promises all the answers, the truth, what is right. When somebody decides to break away from the herd, it is worse than saying the Emperor has not clothes, it is screaming that everybody is naked regardless of who is right and who is wrong. regardless of how they rationalize it, it makes them extremely uncomfortable because it puts doubt in their minds.
Religions build on this mechanism. While they tell themselves they are non-judgmental and forgiving, this only really applies to the in-group, outsiders are of the devil, and god will judge them. They are taught hatred of the anyone outside the group, this is called 'othering' and helps build an us-and-them mentality which keeps them supporting the team and separate from the enemy.
And for anybody in the in-group who cares about you, religion promises them you will burn forever in lakes of fire, so they will worry for your soul and put you under pressure to re-join the herd.
In real life no one will judge you. Depending on where you hail from the faithful might think you are one of them. People will bow at you and say things like: 1; Shalom 2; Salam aleik 3; Excuse me you are Pastor who?
Some people would. To them their creeds are infinitely higher than good deeds. As long as a people deems a certain belief compulsive to salvation, just so long they will persecute and judge to the fullest extent of their power.
When a religion teaches such contemptible filth that no man can tell a truth, or be moral or good, unless he is frightened by god, most followers lose their all senses of proportion and rationality. It is claimed that God had founded Christianity, and that to deny its authority is to be a traitor to God, and thus an ally of satan. To torture and destroy one of the soldiers of satan is a duty most "good" Christian doesn't care to neglect. Nothing can he sweeter than to earn the gratitude of God by persecuting his enemies, is a delight that most Christian never resists.
Why should a Christian be better than his God? Christ taught that unbelief was the blackest of crimes, punishable by eternal torture. Why should the Christian love a man whom her God hates? Why should he show compassion to a kind and noble unbeliever whom his God will burn in eternal hell? Under such benign conviction the Christian judge a living unbeliever unto death in this world, and handover the dead to his god for finishing the "holy hatred in hell".
People who are confident in their beliefs will not judge you; it's only those who have doubts who need to justify themselves by attacking the beliefs or philosophy of others. It's human. Don't worry about it.
Dodgy put it well. Yes, you are highlighting they insecurities of some of these people by showing that you are NOT being punished, are not evil, or "sinful" by not following their particular party line. Don't worry about haters, be true to yourself.