The Swedes have been doing it for a couple of hundred years. Except they go outside and roll in the snow.
If you go to a nicer hotel/spa they often have a "plunge pool" just outside the steam and sauna rooms. It is generally about 5' in diameter and about 8' deep. The water is refrigerated to roughly 45 degrees. Step out of the hot and plunge into the cold.
I think you're OK.
B-