If he has kids I understand not just letting anyone meet them etc. Not health for kids, people coming in and out of there life. But generally the woman has custody of kids. So, when a man doesn't invite you back to his place, generally means there is another woman there. By the choices I have given you, you can decide which one of these guys you are dating. I would say by what you said, you need to drop him.