I would say that SARMS do not have any effect on sex drive - neither positive, nor negative. Therefore, you should be fine.
Regarding the suppression of natural testosterone production, they are indeed suppressive, but minimally so - their suppression rate comprises only 5-10% of that seen with steroids.
If this issue got you really concerned, you can combine SARMS together with HCGenerate, which will keep your libido sky high all the time.