Everything at college is hard if you are not interested in it. It doesnt matter if its physics, physiology or even stupid degree like economics - you will have to spend more time learning stuff than you did in high school, and most things you will learn are useless.
And thats why you have to have minimum interest in what you are doing.
You may think physics is interesting and shit - but you wont be discussing quantum particles and theortize how world was created, you will study tons of math formulas that you dont know what are used for and then you will learn basic concepts of few physics disciplines.
Thats all. Want more? Need to study alone or go for master degree/ phD.
But I would love if you remember one thing: whatever you picked dont ever drop it. End you bachelor at any cost. Remember you dont have to study same course if you will go for master degree. Get Bachelor in anything. IF you picked physics - go for it and dont drop out. You may think that other degrees are easier, but they are not. Get your bechelor at any cost and dont drop out EVER.