Aw erm, I do think that to some degree all men are viewed as intimidating, at least by women and children.
This bothers me too as I'm a real softy, lol
I feel like it's one or the other you're either so weak that nobody fears you (as in the case of women and children) or you have some degree of strength but have to deal with being feared by quite a large percentage of people.
As for ugliness... I don't know. I mean, I think if someone looks particularly unattractive then others might be taken aback, but I'm not sure if that typically is a cause for fear as such. Possibly it increases it a bit over an average looking or attractive man.