My ds is 5.5 y/o.
I think he is probably fine reading the first one. I adore the Harry Potter series, and in fact, have let my kids watch all of the movies, even - gulp - the PG13 one, which I would NEVER do with another series. I would suggest that you read it first and make sure you are comfortable with it. This is what I did, not expecting to like them much. Then I got totally hooked! Who knows? Maybe this will happen to you too and then you will have something wonderful to share with your son. :)
When I was teaching I read this aloud to my second grade class.