I have no experience with things like this but I would have to say follow your heart and do what feels right to you. If you think it feels right to be with this woman then do it. People will judge you but honestly don't listen to them because they don't matter. It's your life and you can live it however you want to! Good luck and I hope everything works out well!