I've been friends with this guy for about 3 years now. We were introduced to each other through a mutual friend who wanted to set us up three years ago. I remember meeting him for brunch but he never really followed up after the brunch date, so just assumed he wasn't interested. Somehow we managed to continue to be friends. He actually ended up liking my best friend but she wasn't interested in him. I wouldn't call him my best friend we do hang out often. About a year ago when I was dating someone else we had gone out with a group of friends and I remember getting food at IHOP after a night of drinking. He started telling him that everyone always asks him why he and I are not dating and that he has thought about going out with me. I might have even said that I have thought about that as well but the timing wasn't right because I was dating someone else at the time. He tried kissing me at the end of the night but I was sober and turned away. Two days later he called and said that I should forget everything he said while we were out the other night at IHOP because he didn't really mean what he said. I was pretty upset, not because he took back what he said, but mostly because he was such a jerk about the situation. He apologized a few weeks later saying that he didn't mean to hurt my feelings and he hoped that we could look past this situation and be friends again. So fast forward a year later.
As Bruce Springsteen sings, "You can't start a fire without a spark." If you feel so unsure, it's not a good sign. You've already spent a lot of time with him, long enough to know whether there is attraction. I just don't see the point in dating someone to whom you are not sexually attracted to. It doesn't mean the two of you aren't wonderful people--maybe there just isn't a spark.
I would be very upbeat at your coffee date and just say that you want to keep it as friends. Let him save face as much as possible.