Of course, anything you do with your body is physical. Social here means you can't do it alone (like sex ) and need another person for it.
You're underestimating the impact of the halo effect and its polar opposite, the horn effect. If you're uglier, people are more likely to see you as dumb, incompetent, lazy etc. (horn effect). This naturally will make forming friendships more difficult (but yes, not impossible). Take very good look at friend groups where the average of the group's looks is low. It's uncommon to see good looking people hanging out in public with ugly people (they'd have to be longtime family friends or something like that).
If someone can't get sex due to their social skills, it wasn't in fact their social skills that was the limiting factor, but their attractiveness. This is
101. Sure, in the extreme cases, like with autism, your inability to socialize "normally" might get in the way of your sexual success. However, in the extreme cases again, where the guy with autism is very good looking, his autism won't always be the limiting factor,
which means looks trumps in the end.