I'm not ripped, but I have the genetic shoulders of an ogre. People in person are generally somewhat nice to me now (when I was a child it was different), but it must be rough having a smaller frame. I bet people are a lot more mean to them and think they can get away with more bullshit just because they're smaller sized.
When I used to box, the biggest threats of mine were smaller framed people since they were faster, and had more stamina due to less oxygen being needed to transfer to their whole body compared to me. A weight example is me fighting a 150 pound guy when I'm 190. I would hit harder, but gas out a lot quicker and was slower. In ji jitsu, my biggest threat was taller guys who weighed more than me.
Being a smaller size in a fist fight is better, but being larger in size is better for when the fight goes to the ground. Sorry for the non-topic information, thought it might be somewhat useful for you. TLDR: I wouldn't focus on just getting muscles, I'd recommend going to boxing gym so you'll learn how to fight but also get ripped at the same time.