It may be your cat is not happy inside or he is bored with staying indoors. Provide plenty of toys to your cat and using toys that need you to intermingle with the cat will help you to bond with the cat. Ensure that your cat can have a view of outside through a window and provide him with scratching surfaces.
Pay attention when going in and out. He doesn't leave unless someone gives him an opportunity to do so. Unless of course, he can open doors like that Russian cat on Youtube. You can leash train your cat and take him out with you, build an outdoor play area for him, or just don't leave him alone when he is outside. I have three cats who live inside. The are all adopted, all front declawed. I trained them to come out on a leash first until they learned where home was. Then I taught them to stay with me when we went out together. They don't go out unsupervised. I live in the city and they aren't able to protect themselves here. You can train your cat too of you are patient and consistent.
Consider finding him a companion so he's not so bored. A kitten, preferably female, so they get along. Cats that spend their time outside rarely live longer than 2-3 years. Just too many dangers out there.
I have 3 cats and live on a farm, all 3 have been doing fine for about 5 years. They are always outside, they kill lots of mice and rats. They're very hardy and go missing for days at a time, but always return. They have no problem with people but do seem to be a little wary around them. I honestly think they have a better life just chilling outside than it would be for them inside. You should be able to let your cat out for awhile so he can go do cat things. But, call him by his name a lot and try to feed him yourself.