Relative Saliency and Ranking: Models, Metrics, Data and Benchmarks

Saliency Benchmarking: Separating Models, Maps and Metrics

The field of fixation prediction is heavily model-driven, with dozens of new models published every year. However, progress in the field can be difficult to judge because models are compared using a variety of inconsistent metrics. As soon as a saliency map is optimized for a certain metric, it is penalized by other metrics. Here we propose a principled approach to solve the benchmarking proble...

Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Extended Abstract)

In [Hodosh et al., 2013], we establish a rankingbased framework for sentence-based image description and retrieval. We introduce a new dataset of images paired with multiple descriptive captions that was specifically designed for these tasks. We also present strong KCCA-based baseline systems for description and search, and perform an in-depth study of evaluation metrics for these two tasks. Ou...

Oracle Berkeley DB: Performance Metrics and Benchmarks

Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

The ability to associate images with natural language sentences that describe what is depicted in them is a hallmark of image understanding, and a prerequisite for applications such as sentence-based image search. In analogy to image search, we propose to frame sentence-based image annotation as the task of ranking a given pool of captions. We introduce a new benchmark collection for sentence-b...

Metrics and Benchmarks for Pervasive Computing

Lord Kelvin (1824–1907) made fundamental contributions to many areas of science and technology. Our modern unit of temperature on the absolute scale (degrees Kelvin) is named after him. His theoretical work on submarine telegraphy enabled the first commercially successful transatlantic telegraph system in 1865. This reduced roundtrip communication latency between the United States and Britain b...