For smaller speakers, polarity rally doesn't matter. But, when it comes to big, multiple speakers being hooked up on the same system, polarity matters a lot. When you ignore polarity in this case, the speakers cancel each other out, which results in low volume.
Yes it matters. The magnet needs to move the cone in the right order. Most pairs of wires, one side will me marked somehow, so at the other end you know which is pos or neg. one side will have grooves or a stripe.