I made very good gains naturally when on SSRIs, so I don't think they effect test levels much. The whole lack of libido and general emotional numbness is probably what you should be more concerned about. The scary part is that you become sort-of dependent on the way it desensitizes you, and you find yourself afraid to come off them. They also seem (at least in my experience) to not interact well with CNS stimulants (ritalin, ephedrine, tren etc.) When I used these drugs while on SSRIs I got very uncomfortable feelings (sudden thoughts of suicide, paranoia, etc.) I stopped the SSRIs against the advice of my doctor (actually had to stop seeing him because he refused to take me off them), and since then have felt much much better (and had no bad effects from the afformentioned drugs).
Again, I am very biased because over the 6 years I was on this stuff I had only negative experiences. I think these medications are over-prescribed to a disgusting degree, and whenever I see ads for them on TV I get pissed off.
Look, through evolution humans have been bestowed with the ability to get depressed because it is a survival mechanism. Just as pain is there to tell you to stop doing something or you will hurt your body, depression is what tells you to change your situation because it is not healthy for your soul. In effect SSRIs are being relied on as a painkiller for the unhappy and unfulfilled mind.