Give it time as people get to know you! Give people "helpful" votes and they will see this and your answers and they will come to get to know you! Be patient and before you know it, you will have more that you can handle ! Just keep giving good answers and comment to people when you like their answers too ! They'll come !
Give good answers, ask good questions, have a good profile, be nice, don;t beg for followers, and you'll get them. Oh, and be patient. Also, Ask.com is not a competition. It doesn't matter who has the most followers. It's no a popularity contest. Remember that, because some people don't.