How can this be an issue unless it's valid? Little kids think everyone of the opposite sex is a BF/GF. Your husband should be laughing hysterically if he has no reason to be concerned. Tell the kids the mailman is not mommy's" BF" daddy is.
explain to them that the other person was a friend and they need to know what the difference is between a husband and boyfriend is. then with some more digging your real husband will understand your not cheating.
After reading some of your replies to some of the answers, all I can say is, are you sure you want to live with someone who is more than likely going to accuse you of something for the rest of your life. You can suggest marriage counseling which would really be a step to getting him the help he needs. He should not and cannot continue to falsely accuse you of cheating and bringing your children into it. You have to think about what he is doing to your children. Using them as pawns is not going to get better. I wish I could tell you something to make it better, but honestly I don't think there is.
Why would the kids say you have a boyfriend if you dont? Why would they say they seen him pulling up his pants in the bathroom, kids that age dont think of things like that by their selves, If he is putting it in your kids head, they you can bet he is probably cheating on you and want it to look like its your fault.