There is a famous study done on monkeys and marijuana. One monkey went clinically insane or something... a wierd outlier... but the only long major term side effect noted was on motivation, the inhibition of it.
For me, weed makes me f up my clean diet, and makes feel like I am not in 100% control of my situation, which is a feeling I find I don't quite like.
For many lifters, it is a great, even essential, part of their lifestyle.
If you enjoy it, if it doesn't stop you from reaching your goals, and you can fit it into your lifestyle, there is no reason not to do it.
Personally, it's just not compatible with my goals in the gym and in life, so I keep it to all but the rarest of occasions, hopefully one day even less.