If you cannot get pregnant it could be from either end. It doesn't necessarily mean it's just the man. However, if you feel that it may be from the man's side, you can always just ask him if he's had one. Also, try going to see a doctor to see if there is anything else preventing you from getting pregnant. I hope it all works out for you in the end.
If he hasn't had a vasectomy there is a condition that men are sometimes born with where they have an extra x chromosome or something to that nature (I'm no doctor but I've heard of it) that makes a man sterile from birth. In order to find out you'd have to have a test done of course. I hope this helps.