Mr. Answer Man says: Well, yes, they matter -- a significant difference in age is at once a great part of the attraction and a source of much trouble. But Mr. Answer Man is guessing that you're wondering what goes through a guy's mind when he involves himself with either a much younger or much older woman.
It's embarrassingly simple.
An older woman is much more straightforward about her sexual appetite and much more appreciative of a younger man's, oh, let's call them jackrabbit intensities. Younger women can be relationship obsessed and tend to constantly look to the future, wondering where they and the relationship are going. Older women live much more in the moment, and their confidence and competence are totally appealing.
Sex and a nubile body.
And that's not all. A younger woman makes a man feel worldly and experienced, sexually as well as socially, which is intensely flattering. (And I'm sure we can all think of men for whom it seems to have become addictive.)
Now, obviously, this isn't all that's going on, but Mr. Answer Man has to be honest -- it's a large part of the draw.