Bit more thought would've been enough to realise evolution in itself is built on trade-offs: a larger body is far less energy efficient and requiring complete different biomechanical structures in order to function well, that's why risks of bone failure and cardiovascular strain directly correlates to human size... So many other issues like body temp and health risks just makes it unfeasible for survival scenarios spanning generations, such an gene would die off quite quickly; especially when you also consider that all these risks would make them even worse in real combat since the human body is just not built for that at all, and that should be obvious from real-life extreme cases of the tallest individuals in history.
There's also the plain fact evolution isn't some profound being overlooking the world, it's just what sticked around after random mutations and worked, rest died out by natural selection; there is imperfections which stuck around in our gene pool simply because of luck and ancestors in safer environments being able to reproduce more easily, it's not going to be perfect.
Objectively, species today are nowhere near as optimised as they biologically could be but it isn't surprising given that evolution doesn't actively aim for 'optimalisation' but just good ENOUGH to survive and reproduce, though attraction acts as a natural filter to weed out obvious deleterious alleles and go for a general trend at least.