Try using a forsener bit, of the proper size. Due to its design it cuts the hole cleanly. Also have a back up board behind the piece you're drilling, as the bit starts to exit flip the board and finish the hole from the other side.
You probably have a dull bit, and the piece needs to be clamped down with another piece of wood UNDERNEATH your piece. Try using a "brad point" precision bit, or a double spur auger bit, a forstner bit will work also.