There isn't necessarily anything wrong, but it is worth getting yourselves checked out. If you spent many years on the pill it can make it take longer to get pregnant. If you think about what a miracle it is for everything to work out to get pregnant it's amazing it happens for anyone. Keep trying your time will come, if not there is always adoption, but I wouldn't worry about that yet, just nice to know there are other options. Best of luck, God be with you :)
You and your husband should visit a gynecologist so as to determine when you are ovulating which will make it easier to hit a fertile period. You should also try to increase the number of times you have sex to at least three times a week. Check out http://www.mayoclinic.com/health/how-to-get-pregnant/PR00103 for more details on how to get pregnant.