It could be because he doesn't trust them to be hygienic, most especially when he doesn't know where they have come from. He once said in an interview that, "In restaurants, when I don't know where the spoons have been, I can't use them. But I can use the other entire cutlery.'
He's afraid of spoons because he doesn't know where they've been before... but he isn't afraid of his own spoons, since he knows where they've been, but if he went to a restaurant the would be scared of the spoons there, since he doesn't know where they've been.
And I've also heard his pet turtle choked on a spoon..??