Don't feel guilty! At worst, get him acclimated to the ear plugs, then you can sign him back up again. Not to mention, summer is right around the corner, and you can take both kids swimming.
I felt the same pain for Amelia. Under 3, our pool only offers Mom and Me class - well, she is so good in the water, she really doesn't need a "Mom and Me" play type class. She needs a learn to swim class. So, we are waiting until she is 3. Boy was she mad that her brothers got swim lessons, and she had to wait. I had no choice - DH was working during lessons, so we go to sit there, while I tried to keep her from jumping in the pool. It was great fun!! LOL.