It is okay to call the cops if the father will not let you have him. However, you should have with you a court order showing that it is legal for you to do so otherwise they will not help. Try speaking to a lawyer about for more advice.
Depends on the custody and visitation agreement. If you have legal custody and his visitation time is up then the cops might do something but if its your exs legal visitation week/weekend then you probably can. If you have shared custody or your ex is has legal custody then you could be the one to get in trouble. If its his visitation time then you cant just go end it when you feel like.