For me, it’s because it’s extremely difficult to talk to someone when they legitimately know much, much less than you do about nearly everything. Politesse only takes you so far — there has to be some possible connection.
What most people believe and “know” is some agglomeration of urban legends, myths, lies and half-lies they saw on CNN (or worse, Fox) and so we aren’t even standing on the same foundation. There is just nothing to talk about besides the weather and that gets old quickly (though I am not opposed to small talk).
Beyond a certain level, talking to most people is just painful for the reasons mentioned above. They are interrupting more important things I could be doing. And in my experience, about 99.5% of people fit this profile, so my lack of social desire has grown from this observation.