This is similar to a situation I've been in myself.
You're right that the crops that get the most attention yield the most fruit. You've both decided that this isn't the kind of fruit you want, so you're watering other crops.
But... You're continuing to tend this crop with him, even minimally, and by doing so you are holding onto an emotional investment in him. This isn't really a "friendship" it's a flirtation that you admit has gotten out of hand but even scaled back, will not fade out completely unless you want it to... And I can tell you don't really want it to. You can't have both an honest marriage and the kind of relationship you're maintaining with this guy.
Texting him once a day is a lot of water for a crop you say you don't really want. If your husband knew your feelings, he wouldn't be okay with that amount of contact.
Unfortunately, you can't maintain an "innocent" friendship once you've crossed the line that has already been crossed. Are you willing to let this friendship fade out so that you can focus your priorities as well? Because from what you say, I think the amount of attention and consideration you are giving this man is too much to honestly say you have "refocused" yourself.
This sounds to me like it's got some self-deception to it. First of all, why would he need your permission to get out of it. That sounds like some kind of attempt to alleviate
....."We are still friends.".....