It's because of human greed and the desire to want what you can't have. There's always something you can't have, and your realization that you can't have it, despite having all of these other things in your life, makes you sad over time. People take the things they already have for granted, including the people in their lives. They get used to a certain level of just having things there in their life for them, so they don't think about how those things make them happy, because it's like eating the same food for the 10000th time or watching the same film for the 100th time. Add to this the fact that most of these materialists have no self-assigned purpose in their life (or at least the internal perception of a purpose that they're fulfilling), and you have a recipe for unhappiness.
Once you become rich and/or famous, you'll begin to see the pitfalls and downsides. For example, you can never be sure if people are friendly to you because of who you are and what you have (sycophants) or if they're genuine. You will always be suspicious of people getting close to you who are from lower social strata than you. Then there's the fact that everything you do in public gets scrutinized. If you're just plain rich without the fame, your own social circles who are not rich look at you with different eyes. They're more critical of you behind your back, and they're obsequious in front of you. They invite you to parties and seek your favor. They kiss your ass and seldom try to hide it. You never know who your true friends are until controversy or failure kicks you in the balls.
You have to look at your position in life as relative to another. Unless you're the absolute king of the world or have the most depressing human existence (both extreme ends of the spectrum), your lot in life will always in relation to another in such a way that you don't truly know how well you have it and can't quite see it objectively.
tl;dr: It does make sense, but from his perspective.
