If the person let you go scot free, then you don't have to worry about it, although the person can still accuse you. He probably let you go because it would have been considered his fault; the cops might have said that he rear ended you.
It could work both ways just like they said he could tell the police it was ur fault . Like this man at Tom thumb did me he said oh yes it's all good and Bam no good when I got a letter from state farm and a warrant for leaving. Got me pretty good. Just hope your guy wasn't laying on the floor hollering when the state trooper got there