Change your perspective. It sounds more like you are asking for permission than advice. The words themselves, "let go," suggest that you are "hanging on." If this relationship is truly over, then it means both of you have gained what was needed from knowing each other and it will naturally fade away, nothing needs to be done. In the meantime, enjoy how much this person has helped you grow as an individual and remain open to letting others come and go throughout your life. We are the sum of everyone we have ever met so when we meet someone, we are also meeting everyone that person has ever met. This person is already a part of who you are and will not truly be gone. On the other hand, if they will not let go of you...the choice of who is a part of your life will always be yours. Just say thank you for the good times and walk away because everything else still applies. Either way, congratulations on recognizing that it is time to part ways, it is not about letting go, it's about personal freedom, yours and theirs.
There is a possibility you could try showing that person how much she/he annoys you so that afterwards he/she cannot blame you for breaking up with him/her. Just try to make excuses if that friend is inviting you or calling you over for some other stuff! Good luck!
If it is a real never give them up. Either talk about what is wrong, Or just leave them alone for a while. I am old enough to know real friends are hard to come by. Yes we all have a lot of acquaintances, and you might be the most popular person around, but true friends no. And it is important you know the difference before you let this person go. It would be interesting why you want to leave them.