It's related to the fear of being shown up as dumb.

People walk into my office,look at the bookshelves and ask "Have you read all those books?" as if there's something odd about having read several hundred books in the course of 54 years of life. The answer, "most of them", never seems to satisfy for some reason.