Links tell us about lexical and semantic Web content

The latest generation of Web search tools is beginning to exploit hypertext link information to improve ranking 1, 2 and crawling 3–5 algorithms. The hidden assumption behind such approaches, a correlation between the graph structure of the Web and its content, has not been tested explicitly despite increasing research on Web topology 6–9. Here I formalize and quantitatively validate two conjectures drawing connections from link information to lexical and semantic Web content. The link-content conjecture states that a page is similar to the pages that link to it, i.e., one can infer the lexical content of a page by looking at the pages that link to it. I also show that lexical inferences based on link cues are quite heterogeneous across Web communities. The link-cluster conjecture states that pages about the same topic are clustered together, i.e., one can infer the meaning of a page by looking at its neighbours. These results explain the success of the newest search technologies and open the way for more dynamic and scalable methods to locate information in a topic or user driven way. All search engines basically perform two functions: (i) crawling Web pages to maintain an index, and (ii) matching URLs in the index database against user queries. Effective search engines achieve a high coverage of the Web, keep their index fresh, and rank hits in a way that correlates with the user's notion of relevance. Ranking and crawling algorithms use cues from words and hyperlinks, associated respectively with lexical and link topology. In the former, two pages are close to each other if they have similar textual content; in the latter, if there is a short path between them. Lexical metrics are traditionally used by search engines to rank hits according to their similarity to the query, thus attempting to infer the semantics of pages from their lexical representation. Similarity metrics are derived from the vector space model 10 , that represents each document or query by a vector with one dimension for each term and a weight along that dimension that estimates the term's contribution to the meaning of the document. The cluster hypothesis behind this model is that a document lexically close to a relevant document is also relevant with high probability 11. Links have traditionally been used by search engine crawlers only in exhaustive, centralized algorithms. However the latest generation of Web search tools is beginning to integrate …