Apparently, it's because it's most people's better side of the face. I got my own pet theories, though.
We write left to right, so it might engage the parts of our brain that are associated with the language that we also write from left to right to think of words to describe the person we're looking at. I also think that it might engage the right (emotional) side of our brain more, which makes more in tune with how we feel about the face we're looking at.