If you do have low self-esteem, people can sometimes sense it. And, of course, being shy can sometimes come across as aloofness. (I have BOTH of these problems..........agh!)
Whenever I meet someone who I think would mix well with me, I try to speak with them on private terms. If they feel even a slight connection with you or that you're paying attention to them, they might be more open to starting something up. For instance, if you know they like a band, ask them about it. They'll appreciate that you went out of your way to pay attention to them.
Well, in the workplace, I am pretty confident. However, in my personal life, I can be less confident.