Maybe music is as super-stimulating as sugar and pornography, an artificial abundance of something that is naturally scarce; most people are addicted for life after just a handful of exposures, becoming miserable if they are deprived of music for an extended period of time.
A while ago I watched a video of a Buddhist monk where he talked about giving up music, and how it was tough, but he no longer misses it. His (and other) religious traditions view music as a form of hedonism that ultimately hinders spiritual development.