Being a single-digit millionaire doesn't even qualify you for private banking at many larger banks, I know JP Morgan requires at least $10M of your assets under management for them to sign you as a client. If you are a city-dweller and you are a middle-class homeowner than most of your net worth is tied up in your place of residence, your lifestyle isn't glamorous, you don't have a lot of spare cash to throw around for vacations and bling.
Obviously, if you have $2M and you're in flyover America, or Eastern Europe, or South Asia then that's enough to live a very comfortable life. But $2M in NYC, London, Hong Kong, Geneva, Monaco? Not so impressive.
Don't get me wrong, being a single-digit millionaire isn't bad, especially if you have no dependents. You won't have to worry about healthcare or other basic needs. But you don't have the global mobility that the wealthy do in terms of being able to get investment-based visas and passports, buy second and third homes, expanding your social circle through philanthropy, et cetera.