Because they're surrounded by large volumes of stone, it takes a great deal of heat to raise their temperature (they also cool slowly / the stone has a high thermal inertia). As you go deeper, the temperatures generally fluctuate very little, and reach an average temperature close the the average (year round) temperature of the local area. As they tend to be moist, there may be some evaporative cooling that lowers the air temperature a little. Also, as blacnblond04 mentioned, sunlight doesn't tend to warm them.
If you go below 10-15m underground it's about 12 degrees Celsius, almost everywhere in the world. The surface temperatures cannot take influence so deep, and the heat from the inside of the planet doesn't get up so far. This can be different in local spots, of course, with volcanic activity or so. If you go deeper, it gets continuously warmer. At 1000m its close to 30C already.