Of course I don't have any magic answers, but flax appears to be an even bigger source of estrogen than soy, and even more dangerous. Here is an interesting article. The author does not cite any sources.
http://totalityofbeing.com/Articles/FibroisandIndustrySpin.html