There have to be theoretical limits to any track/field/swimming event. Butmy answer is "it depends".
The question you're asking though - records will never cease to be broken. The margins by which they are broken should probably decrease, although as the body becomes better understood, performance and supplementation will improve.
Roger Bannister's 4 minute mile is nothing now, the record is like 3:43...in less than 50 years. In 50 more years, however, is a 3:20 possible? CAn human bones and muscle do that?
Consider also that Mark Spitz's 7-gold performance times would now not win a high school meet; likewise Jesse Owens.
However, in 1988, Ben Johnson won the gold (on steroids) with a 9.79. That smashed the then-record 9.93. Today, the record has been lowered to 9.79 (legally). This is an example where you are seeing asymptotes - a 3:20 mile will happen before a 8.5 second 100m dash.
Records will never stop being broken. The margins will shrink. In some events we are already there.