That really sucks.
So many people walk around only thinking about themselves, and only look at the median average as "the other people." But rarely do they take the time to empathize with individuals who have truly unfortunate conditions.
They look at you and think, "Damn, sucks to be short!" But that's not the story for you; you wake up in the morning: you're 5'2"; you go to work: you're 5'2"; you go out to lunch: you're still 5'2"; you are 5'2" every moment of every day until you die. You have to deal with your height deficiency so often, that it can come to dominate your life, especially all social interaction. Big guys get automatic respect from everyone -- people just naturally defer to their opinions. Little guys really are "less of a person", they are a smaller part of their peers' lives.
Small men may not matter less, but they are treated as less.