Your sex drive has nothing to do with your misogyny and it doesn't lower it.
Let me put it to you this way, someone hating animals has nothing to do with them liking the taste of meat. They are two completely separate things.
That person may hate animals because they see them as "annoying" or "dirty", but that hatred would never "carry over" into a biological craving to where they wouldn't want to eat meat, it doesn't work like that, those two things aren't connected.
Same thing with sex and women.
Someone hating women doesn't mean they will enjoy fucking women any less. If anything your hatred for them might make fucking one even more enjoyable because you know they don't want to fuck you.
Someone hating cows doesn't mean they will enjoy eating burgers any less. If anything your hatred for them might make eating one even more enjoyable because you know they had to die for you to eat them.
But you'll notice that this is because the "dynamic" between the two acts I described is a relationship where the "hater" is the predator and the thing that is hated is prey. Maybe in the case of women their hatred of men would affect their attraction to one, but that's because they are the ones that are "acted upon" (prey) rather than the "actors" (predator).
There's a huge difference between the experience of being a predator towards something you hate (feelings of pride, dominance, power, etc), and being preyed upon by something you hate (feelings of self loathing, submission, weakness, etc).
Sex is inherently an act where men dominate and women submit, so feelings of hatred will make likely enhance the experience positively for the male, but in the case of the female it won't.