So many women would love that in their SO. Just goes to show that there is someone for everyone. And, clearly, he's not right for you. Do both of you a favor and cut him loose. The sooner the better.
Do both of you a favor and break up. "I decided to give it another try" is not enough reason to get back together. Sounds like you got lonely after a few weeks without a date. Only acceptable reason to get back together is "I missed him so much and realized I could not imagine living my life without him". If the feelings are not that strong, then they probably aren't enough to outweigh whatever differences motivated you guys to take a "time out" in the first place.
You either aren't into sex or aren't into him. Either way, you and he are both better off apart. Even if you get a little lonely sometimes. Spend time with girlfriends. Volunteer at a homeless shelter or soup kitchen. Don't go back to him.
When you see it coming, duck!