Nah, baseball pitchers are big and fat, they just need to be able to generate momentum with weight, long levers, and apply as much slingy Ness as they can.
Cricket bowlers tend to be very athletic yet strong, and normally tall, because they have to bowl with a straight arm, and use the ground as well as their grip to make the ball move to spoof the batsman. Because they can't sling their arms much, they generate speed by running fast.
Baseball hitters are brutes because it's all about having enough power and momentum to hit against the momentum the pitcher threw it with, but cricket batsman in some cases are big wild sloggers, and all are normally strong, but it's much more about technique, and using the balls momentum to flick it around, plus they sometimes have to run all day, for 2-3 days!