Details on Biasing Web Search Results for Topic Familiarity

A typical web search engine returns a mix of introductory and advanced documents (around 50%) in response to a random selection of queries. Depending on a web searcher’s familiarity with a query’s target topic, it may be more appropriate to show her introductory or advanced documents. We conceptualize the notion of introductory and advanced documents in a way that obviates additional user-interaction and changes to existing search engine architectures. We show that topic familiarity required to understand a document (familiarity level) is a notion that people can agree on, as borne out by high inter-rater agreement (70%). We also show that this familiarity level is not predicted by reading level, so new methods of identifying it are needed. We develop a method for biasing the initial mix of documents returned by a search engine to increase the number of documents of desired familiarity level up to position 5, and up to position 10. Our topic-independent and user-independent method involves building a supervised text classifier, incorporating features based on reading level, the distribution of stop-words in the text, and non-text features such as average line-length. Using this familiarity classifier, we achieve statistically significant improvements at reranking the result set to show introductory documents higher up the ranked list. Our experiments indicate that we can perform this search result biasing for arbitrary users on arbitrary queries.