Here's a good example of the snobbery associated with this book. Implying it was "difficult" like it was over my head. It wasn't difficult, it was tedious, boring and meandering.
It's not a good example -- he's implying that you're irrationally upset over not liking a book while other people do like it, not that the book itself was too difficult for you.
If this is what you're interpreting as people talking down to you / claiming that you're too dumb to understand a smart book, you may want to consider that it's a you being overly defensive thing. You're allowed to like what you like. If some people have been dicks to you about your taste, I don't see how that has anything to do with this book. Forget those people.