It depends on the question. If it's JUST a question: Is it or is it not? I find it makes no difference. If you mean a request, to give or not give permission or an item, it is usually at least a little harder to say no.
It depends on the situation. "Hey, you want to jump off a bridge and potentially kill yourself?" Obviously, I'd say no. "Want to watch a stripper?" The right answer would be no, but it would be hard to say it.
Absolutely not. After years of torture from saying yes to some of the most stupid requests... I do not have an issue saying no. I do think about it though before I make my decision, but once I've made it... then I have no problem saying it.