Pure physics, classical mechanics, to be precise. Imagine how tough it would be to do a lateral raise if your arms were 3 inches longer (a lot!), now if you multiply that through every joint in the body, a 6'2" gymnast would have to fight against double the force across some joints when completing his routine in comparison to a 5"2' guy with the same proportions. Not saying some tall guys can't do it, but this is the olympics, so everybody has the ideal body, all the swimmers are tall, all of the high jumpers are ectomorphs, etc.