As a responsible parent, you should know that your son needs positive relationships with extended family, even if they are crabby to you. Your son doesn't need to know the nasty bits. You know they are nice to him. Let your son have time alone with them.
You should let your son decide if he wants to have a relationship or not with them and draw his own conclusions even though it isn't right they are mean to you your relationship with them doesn't have to impact his and it doesn't sound like he is having a negative relationship with them. Have you tried to talk to your husband about it? Maybe he can help? I think there are more possible options for a peace treaty or resolution if explored further maybe.
Yes it's bad. If they want to see the baby and will be loving grandparents, then let them see him. Many divorces are nasty and in-laws become cool to the other spouse. Doesn't mean they don't love the grand kids.
Using a child as a pawn to control and manipulate is never a good thing. Just because you have a problem with your extended family doesn't mean that this child should grow up not knowing his family. That is just not okay.