I'm the same as you, but that's because I literally grew up on the beach in New Zealand, and swum nearly all year round. When I was older, we moved to the Gold Coast, Australia and I surfed nearly every day for my entire teenaged years; work gets in the way but I still surf as often as possible.

The one thing though, is that even with that time in the water, not having a lifejacket on a boat is absolutely foolish. The cold water really does shock your body into immobility, no matter how good of a swimmer you are. It's scary as hell.