Swimming in actual street clothes is a for sure fire way to slow you down in the water, the weight of the clothes once they are wet will slow you down big time. Also, the wet clothes will limit your body movement making it even more difficult to swim. You absolutely want to swim in a swimsuit, regardless of if it is speed brand.
Swimming in street clothes would definitely slow you down more than a pair of Speedos. Clothing would adsorb more water therefore adding additional weight for you to swim as well as constrict your arm and leg movements, also causing slower swimming.
To add additional thoughts... if you swim in a wetsuit, such as those used by triathletes, the material used in them will provide additional buoyancy (and warmth). Clearly, you don't want to swim in clothes as the others have mentioned. I had to do that for a survival swimming class, and it definitely makes you work hard to stay afloat for long periods.