My husband and I have always had issues about sex.
I'm sorry for what you are going through.
It is not possible for me to say whether or not it is too late for the two of you. That is something you both need to decide. What I can offer is my experience with what I call "distancing- emotional and physical".
I too have gone through times when I no longer offer my affection. In fact I would purposely stop myself from doing what comes naturally to me. I didn't offer hugs, or pats on the knee- or even smiles! I refused to engage in conversations or share my day to day experiences with him. I admit it is not the best way to behave all things considered, but it is a self preservation sort of thing. The distance makes his rejection hurt just that little bit less. Trust me I am now trying my best NOT to react this way. It only pushes us further apart in the long run and it is not who I am. I am naturally a very affectionate person.
I can't speak for your husband but perhaps this is his way of dealing with your ML?
Sorry I couldn't be of more help.
Hang in there.