It's not a bad idea to take a pregnancy test just to ease your mind. But, it could be something other than pregnancy if the test comes back negative. You still want to check with a physician if the test is negative to see why you are going to be bathroom more often than normal.
I'm not too sure if it's the same as every one else, but with all 4 of my kids, I got pregnant during the week after my last period day. So you'd might want to calculate when you'd ovulate and if you were ovulating when you were having sex, then yes a pregnancy test might be a good idea. But you won't actually start peeing more till the baby is large enough to prevent the bladder from getting full like a non pregnant woman.