In past times they were not worn together but now it is OK. Just like matching your shoes to your bag or belt--it used to be yes but now it's no. It seems like non-matching is the new style. But somehow it's all supposed to "go" together.
I say they DO go together, and look nice... Unless you're talking about leather. A brown leather belt & black leather shoes -- nope, that doesn't work. I like all my leather items to be the same color, but a brown skirt or pants and a black top look fine together. :)
Heres what I think: I say dress the way YOU like, wear what makes YOU comfortable. I have never been one that thinks you have to be fashionably correct and still look good. I think some of the "styles" are horrible.