Not at all. Visit at break time or lunch, this may give you a chance to look around and not be in the way. After hours would be better and you may be better able to see the progress that has been made. Once the main structure has been built, visit more often so you can see the detail work going into the interior fit out.
Visiting is one thing and asking questions is another. The job supervisor technically is the person you should ask questions regarding the construction. The other workers or trades mostly will not have complete information and are sometimes told to direct all questions or concerns to the general contractor.
No what is wrong in it..It is your plot and you do have rights to visit anytime.As you should have knowledge that how much your house is built and how construction is going on so for this regular visit is must.