I can understand your hesitation, you can ether wait it out or dump him. But understand his point of view, he is thinking that he doesn't have long to live, so why not a a quick fling. It's not right, but he probably nit thinking correctly right now. Putting everything else aside. I sure he loves you very much. You guys need to sit down and have a serious talk to clear the air. good luck to you both.
There does not appear to be any real love or respect left. I don't know your circumstances or concerns but you need to take control of the situation and make it work for you. I would end it if you have others to support you.