Well for one perhaps you should stop letting this person in to use the phone if you have been doing so. Do a little investigating to find out who this person is and where they come from; trust your instinct. If you are getting a negative vibe there is a chance your vibe is right. Tell them nicely to please stop coming by to use the phone or just stop answering the door. If they continue it might be time to get the police involved.
They are possibly using that as a excuse to gain entry to your home to see if you have anything worth stealing. Never let people that just come up to your door randomly, come in to your house. Don't be afraid you're being rude; it's smart to be cautious and protect yourself. Also, if your are female and alone, you may get yourself raped letting strangers in like that! Please be smart!