Depends, what college? If you go to a party school there will be lots of frat and sorority members and they will be everywhere, and they are annoying. At other less degenerate colleges it will be not so bad. But as someone 30 or older they won't want anything to do with you either way, even though you may see members all over the campus.
You wont be able to avoid them, especially on the weekend when they are walking from place to place to party and drink.
If you are getting a masters it doesnt matter anyways because frats and sororities are usually just for undergrads who live on campus and people leave once they are done with their bachelors degree.