Long time since physics classes, but I'm agreeing with Answer_Otter. There is a bit of friction, but mostly it's from the kinetic energy of the hammer getting converted to thermal energy, I do believe. Both nail and hammer will eventually get warmer as the kinetic energy of the hammer swing rebounds between them. Then there is also the slight compression of metal molecules which would also release heat, if I remember rightly. This compression would emit some heat. For another example of heat from compression, I know that in an air compressor in an air conditioner, the air is compressed and so it throws off heat (which is then carried away by some mechanism). When that compressed air is then released and expands again, it is much much cooler. Compression would do the same to liquid (pressure cooker) or metal, but metal takes a lot more energy to compress. I think that's it, but feel free to educate me if I'm wrong... :)
As the others said, friction likely plays a part. However, there's something kind of complicated going on at a molecular level that the term friction may not adequately describe. When you hit a nail with a hammer, even if the nail doesn't seem to move, the large quantity of kinetic energy being transferred to it by the hammer has to take some form. Some of that will be movement, some will be sound (vibration at a frequency that you can hear), but a lot will be heat (in a sense, also vibration). I don't exactly know the mechanism by which the compressed/distorted metal gives off heat, but I think a term other than friction is likely appropriate.
It's all from the friction created by the resistance of the wood being nailed. The hammer provides the kinetic energy that is 1/2 of the equation. The nail is the transfer agent and the wood provides the equal and opposite directional resistance that causes the transformation of kinetic energy into heat.