Will I hear from him?

I recently decided to try online dating and managed to rule out every guy who contacted me, except one. We e-mailed regularly and chatted online for a few days before he asked for my number. I wasn't comfortable giving it out at that point, but when he asked again a few days later, I did give it to him. He called me this past Sunday, Monday, and Tuesday and we had great conversations for about an hour each time. We seem to have a lot in common and carried on conversations that varied from intelligent and deep to just plain entertaining. During our conversations he made it clear that he's not the type to play games and told me how much he enjoyed talking to me. So, I assumed that we both realized that our personalities are compatible and the only thing left was to see if there was any chemistry.

We met for dinner on Wednesday night and had a great time (or so I thought). Since I have a habit of overanalyzing things, I've been thinking about some of the things that I took as a sign of his interest. He reached over during dinner to turn my necklace around, and when I made a comment about an eyelash near the corner of his eye, he took my hand and basically directed me to remove the eyelash. He even commented on the fact that he'd have to do something about the fact that I haven't been to a certain local place. At the end of the evening, we walked to a subway stop together, and he gave me quite the goodbye kiss and said he'd e-mail me the next day (yesterday). Well, before he had a chance to e-mail, I decided to send him a short message thanking him for dinner and telling him that I enjoyed myself. I made a comment about work too. He responded, but with only a very brief message that was basically a complaint about how busy he was, but he ended the message with ":)", making me think he was trying to lighten up the tone of his message.

I really expected him to call last night, since he had called every day prior to our date, but he didn't. Since I know he normally doesn't have time to e-mail until late afternoon, I know I'm going to be dwelling on this all day. Can anyone give me their opinion? Am I stressing for nothing or will he really not call again? At what point should I give up on him?