Humans are such social creatures and women are treated as such a commodity that I doubt physical toughness is required or even desired (I've tried that route and failed). Status matters because in a town of 2,000 people, a popular guy can rally 100 men to fight with him. A tall guy cannot beat up 100 other males. But still, status does not matter at all, because WOMEN ARE TAKEN CARE OF AND FUCKED BY NO MATTER WHO OR WHAT IS IN CHARGE OF THE TRIBE/CIVILIZATION. If one tribe comes in and takes over, they just take over for the other males. No Stacies were harmed in the filming of this movie (this explains extreme lack of loyalty, as Stacies loyal to new tribe would have preferential treatment, as would their children).
I personally do not believe in evolution, but that's how it would work if looks were so fucking important. It makes no sense from a genetic standpoint and you are correct, but there are far bigger holes in evolution's explanation of female attraction that are complicated and I've explained before.