The one thing you need to realize is that no matter what ex you have or deal with, you just can't be friends, it's as simple as that. The fact that you and your ex are still in contact is not good for you.
And your ex seems like he doesn't know what he wants either, the current GF is a rebound. Think about it like this, if your boyfriend were talking and having lunch with an ex, wouldn't red flags be up on that? I know mine would be.
As far as the whole best friend thing, you HAD that with him, you don't have that with him now. As much as you want to, you have to remember that it was in the past. If he were single and he gave himself time to get over you and the current GF, things would be a lot different with him and his mind would be a LOT clearer than it is now. He's only filling a void with the GF and doesn't know what end is up right now, especially if he's contacting you and having lunch with you.
Remember that you need time to get over a relationship, and you haven't had that yet. You are looking to the last person that you had a connection with, and are hoping and wishing that things would work out. He clearly said that he has issues he needs to work on. Doesn't seems like much has changed with him if he's still in the same job, and still has the same issue. I don't think you are missing much if the current GF is going through that. You said about backing off, and I totally agree on that.
You need to have time to distance yourself from him, wish him well in life and move on. More than likely the timing wasn't right and he wasn't the right one for you. There will be plenty of guys down the road that will have their stuff together and won't put themselves in a position to talk to ex's and repeat patterns, and it will be a lot easier than what's going on now. Good Luck!
I agree with sweetpea here. I am very glad you understand WHY you went back to an ex - Because you were lonely and he was a comfortable person. You didn't have to suffer the uncertainty of getting to know him a second time. And the great thing was you were able to leave.
You cannot get over an ex and remain friends. It is not possible. Many have tried and all have failed at some point. It is possible to be friends again in the future - But you HAVE to break off contact completely first, allow yourself to heal, and often by the point that you are over him, you don't really care to continue a friendship. My best friend is going through something very very similar and it is difficult watching her make the mistake of trying to remain friends with him.
I wish you luck, but if you make the right decisions you won't need luck.
Before you jump back into your ex's arms .. have the issues that broke you apart resolved?
Thank you for the advice.
Maybe I'm reading too much into this...because I ANALYZE....ok, I SUPER ANALYZE things...but it doesn't seem like your "trying" to forget about him...it seems you can't stop thinking about him!