When I was swimming more, they said not to ever clean out your ears because the wax would help... with stuff. That was about as specific as they got.
I have also read that Q-Tips are bad for ear stuff.
That said, I have never ever ever had an ear infection, and my way of cleaning my ears repulses everyone that I've ever told about it (or the few that have seen it).
I have never been to an ENT doc, but I started very much the way Raina said - with a bobby pin - I used the curved end to scoop out the ear wax from my ears.
I then would just spread it on the inside edge of my shirt or boxers.
It felt FANTASTIC.
Then I graduated to the system that I use now. I take the plastic wrappy things that come when you buy new electronics. They are thin flat plastic with a metal wire inside of it - too thin doesn't work, too thick doesn't work.
I don't clean them in any way, I just wrap them around my finger tip and then twist it down so that there is a little scoop at the end.
I then put that in my ear and scoop out the ear wax.
I do this anytime that my ear feels as if it needs it - don't know how to explain it, but everytime it feels that way, I can scoop out wax.
I never clean this thing aside from just wiping it on my clothes (where it can't be seen).
I have never had an ear infection (and I'm really hardly ever sick).
I still swim, and I have less of an issue with water staying in my ear and my hearing is actually annoyingly good (I tend to hear background noise too much).
The hearing thing might be the only problem I can think of for this.
I generally think people are too silly about most germ issues.