Install the app, follow the directions to activate the international keyboard, delete app from iOS device. After you install the international keyboard that the Emoji app includes, you don't have to leave the app installed. I'm speaking from first-hand experience. I haven't had the Emoji keyboard installed since the day I originally downloaded and installed the app and the Emoji icons keyboard is still present and has been through several devices, backups, iOS versions, etc.
If you also want to delete the keyboard once it's installed, you need to go to Settings > General > Keyboard > International Keyboards then click the Edit button and then click on the red minus sign next to Emoji. You can reinstall the Emoji keyboard by reinstalling the app and then following the directions again to install the keyboard.
No, you cannot delete the Emoji free Application after it is hooked up to your keyboard on your iPod. The only thing you can do is to disable the Emoji keyboard because it's usually included in your iPod and the Emoji application only enables it.