As a freediver (I've been as deep as 120 feet. Not on the level of these guys, but not a newb either) I can tell you that panic is not a part if it. You almost have to be in a meditative state to keep your HR slow enough to go any reasonable depth.
I prefer having air, and generally only freedive when spearfishing, but I still understand why. It is a challenge, but also a way to seamlessly blend into the ocean that can't be recreated in any other way. You become part of the swells of life instead of an invasive species. Fish swim with you instead of away. It's amazing.