This whole thing actually started several years ago.
Are you sure he's seeing the e-mails you've sent? If it's going to his message inbox on facebook, it doesn't mean he's checking the in-box whenever he logs on to facebook. He may be like me and be a frequent user of facebook who doesn't normally get messages. While I haven't missed any facebook messages yet, I have missed important e-mails I've gotten to other e-mail addresses for similar reasons. He should be getting an e-mail notification to a separate address when somebody leaves him a facebook message so he knows to check, but he may have turned that option off or not be checking his e-mail address.
I would probably still hold off on inviting him out again at this point anyway and wait to see what he does. You've invited him out twice now, and he hasn't asked you out once yet. If he is really interested, he will make the effort to contact you and invite you. Otherwise, it's possible he may have just been trying to follow good manners when he was out with you. Good luck.
I think you're jumping the gun a bit.
As far as how you described your dates so far, he just seems to be a very good date in general, meaning, nothing stood out as being particularly just because of you.
It's been four years, settle back a bit and quit acting so very eager, almost like you haven't been on a date or had any male attention in the four years you've been apart, while on the other hand, he's been apparently perfecting his dating skills.
Email and text are great for touching base, meaning it should be seen as mainly one-way, but if you need to have an answer to any sort communication with him, do that on the phone. Email and text are THE worst ways to communicate if you really require a response.