It's because the estrogen steroid receptor was arguably the first steroid receptor to appear though evolutionary history, and because it precedes every other steroid receptor, there are a lot of different chemicals that bind to it (at least that's my theory). As long as a chemical has the estrogen chemical backbone, so to speak, it will probably fit to the receptor or have an affinity to a degree. That extends to a lot of different chemical classes.
We as a society are introducing everyone, without their consent, to a broad range of chemicals that have never been in existence ever, with unforseen consequences. It would make sense that these chemicals could have some estrogenic activity to a degree.
Oh yeah, and about increasing your testosterone, don't listen to most of the broscience, and for sure don't try to listen to the mainstream advice for a "healthy" diet. The best way to get high testosterone and to kick estrogen to the curb is a high calorie diet, high carbs (preferably sugar), MODEST protein consumption (protein is pretty important, but you don't want too much of it either), and no unsaturated fats. Also, avoid phytoestrogenic foods like the plague (they reduce DHT and progesterone, and cause your estrogen to get high as a result). Fat soluble vitamins are also especially important, because they are actually quite antiestrogenic and most people are quite deficient in them nowdays. And get more cholesterol in your diet, because all steroids in your body are synthesized from this chemical.