If you are wearing them while exercising or wearing them everyday they will start to smell. You feet sweat and the moisture can allow bacteria to grow causing your shoes and your feet to smell. If you are talking about doing every so often and the shoes can air out and dry out in between wearings it is fine.
I usually don't wear socks no matter HOW stinky I become. If you have the right amount of room in the front and back it is unlikely you will get blisters depending on what you do. The only problem with no socks is easy access for wood chips, sand, mosquitoes, and other icky stuff. But other than that, no socks are just great!