My two cats don't do that. Sometimes, when you are interacting with your cat, if they don't trust you, they will keep their eyes on you at all times. I don't know how long you have had your cat, but he might not completely trust you yet. Just act steadily around your cat, and try not to make sudden, quick movements toward them. Cats are very in-tune with their surroundings. They have the most wild left in them of all the domestic animals. So, they will often act like their wild cousins do. Once he completely trusts you, he will probably stop the action of swiveling his head around.