Many cold and flu-viruses are able to cross the superficial lining, the mucous membrane, of the nose, pharynx and respiratory tract. Then it gets dangerous: These viruses can attack the heart valves and the heart muscle. Repeated infections of the heart can lead to a sudden death during heavy exercise. It might be best to let your body win the fight with the virus and your temperature return to normal before you swim.
If you catch a chill, you will get sicker. However, if you make sure to get warm and dry immediately afterward, it may actually help bring your fever down to cool off in the water. Just get out immediately if you start to shiver.