I wouldn't sweat it as long as it doesn't turn you into a couch potato. I know plenty of daily weed smokers that are huge-many toke up just before working out too.
A long time ago there used to be a belief that cannabis raised estrogen levels-which is outdated 411 that was disproven awhile back.
Now get ready to be flamed by the self righteous AAS users that believe all other drugs besides AAS, boner pills, and happy pills like Prozac are for losers.