I agree with you if by "muscle men" you are talking about a top level amateur or pro bodybuilder builds. However, if you're talking about guys who are big enough that they are obviously look like they're on steroids I will have to disagree, at least based on my experience. When I go out women love to feel my arms, chest and abs while ooohing and ahhhing(in public). I've slept with a lot of women who have never been with a "big guy" before and they absolutely love the way you can sling them around the bedroom like a rag doll while they get to hold onto those hard muscles. I have met women who were indifferent to my size or slightly intimidated but it has never been a "deal breaker."
Also, one thing I've learned about women is to never go by what they say but watch what they actually go for. Every woman I've dated except for a handful would have said they weren't into bodybuilder bodies but they were all about mine. Likewise, I've had countless female friends say how they weren't into ripped abs and then they salivate when some guy on TV selling an ab video shows his ripped abs. Just my two cents.