We study the consistency of listwise ranking methods with respect to the popular Normalized Discounted Cumulative Gain (NDCG) criterion. State of the art listwise approaches replace NDCG with a surrogate loss that is easier to optimize. We characterize NDCG consistency of surrogate losses to discover a surprising fact: several commonly used surrogates are NDCG inconsistent. We then show how to ...

A central problem in ranking is to design a ranking measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the widely used Normalized Discounted Cumulative Gain (NDCG)-type ranking measures. Although there are extensive empirical studies of NDCG, little is known about its theoretical properties. We first show that, whatever the ranking function is, ...

A central problem in ranking is to design a measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the Normalized Discounted Cumulative Gain (NDCG) which is a family of ranking measures widely used in practice. Although there are extensive empirical studies of NDCG, little is known about its theoretical properties. We first show that, whatever the r...

A central problem in ranking is to design a measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the Normalized Discounted Cumulative Gain (NDCG) which is a family of ranking measures widely used in practice. Although there are extensive empirical studies of the NDCG family, little is known about its theoretical properties. We first show that, wha...

This paper describes our approach to the NTCIR-12 MobileClick task. First of all, we do some extra process on the baseline. Next, we try to use a totally different method from baseline which is machine learning. Finally, tune the two types into better situation and apply them to test data. Our system achieves an nDCG@3 score of 0.7415, nDCG@5 score of 0.764, nDCG@10 score of 0.8059, nDCG@20 sco...

The Normalized Discounted Cumulative Gain (NDCG) is a widely used evaluation metric for learning-to-rank (LTR) systems. NDCG is designed for ranking tasks with more than one relevance levels. There are many freely available, open source tools for computing the NDCG score for a ranked result list. Even though the definition of NDCG is unambiguous, the various tools can produce different scores f...

Users express their information needs in terms of queries to find the relevant documents on the web. However, users’ queries are usually short, so that search engines may not have enough information to determine their exact intents. How to diversify web search results to cover users’ possible intents as wide as possible is an important research issue. In this paper, we will propose several subt...

In NTCIR-9, we participate in the Intent task, including both the Subtopic Mining subtask and the Document Ranking subtask. In the Subtopic Mining subtask, we mine subtopics from query logs and top results of the queries, and rank them based on their relevance to the query and the similarity between them. In the Document ranking Subtask, we diversify top search results using the mined subtopics...

In this paper, we describe our approach to linking news articles in a cross lingual environment, English and Hindi, as submitted for the CrossLingual Indian News Story Search (CL!NSS)[1] task at FIRE'13. In our approach, English documents are first converted to Hindi using Google Translate[2], and compared to the potential Hindi sources based on five features of the documents: title, the conten...

We formulate the problem of supervised hashing, or learning binary embeddings of data, as a learning to rank problem. Specifically, we optimize two common rankingbased evaluation metrics, Average Precision (AP) and Normalized Discounted Cumulative Gain (NDCG). Observing that ranking with the discrete Hamming distance naturally results in ties, we propose to use tie-aware versions of ranking met...