Automatic grouping of semantic keywords to improve image rendering

The ultimate goal of automatic image rendering is a system that gives at least as pleasing results as a human expert using an image manipulation program. In this article we demonstrate that the exploitation of semantic image keywords is a promising approach towards this ultimate goal. We develop a keyword classification scheme specifically for the purpose of automatic image rendering. Further on, we propose a method to automatically classify keywords into these classes. We discuss the results based on experiments with a database of 40’000 images, annotated on average by five keywords each. Introduction Enhancing digital images to make them visually more appealing is an important aspect in digital photography. Many software tools exist for this task, but due to the semantic gap – the fact that computers don’t understand semantic context as well as human beings do – they do not work automatically but need human guidance. Let us consider an algorithm for enhancing the attractiveness of human faces by warping them to make the face appear more symmetric [9]. This is good in many cases, but wrong if a facial expression is desired that does not match common standards of beauty (e.g. frowning one’s brow). Unlike a computer, a human being would recognize that the frowning look is essential and either leave it asymmetric or make it even more apparent. In the context of this work, we define image rendering as either color rendering [1] or photo enhancement (e.g. adjustment of color, contrast, sharpness) that is either applied globally or locally to an image. We focus specifically on semantic based image rendering. Thus, either the whole image or different regions of an image are processed according to the image’s or regions’ specific content. Content aware image processing is not a new topic. Cameras exploit user settings for internal processing of images if e.g. portrait mode is chosen or if the user defines the light source for white balancing. Technical metadata can also be used for indoor/outdoor classification [3]. Ciocca et al. propose a system that uses different classifiers and detectors to estimate the content of an image and base further processing on that information [4]. These examples show that technical metadata and automatic classifiers can add some semantic information, although it is very limited and on a much lower level in comparison to the semantic understanding of a human being. A different and promising approach towards automatic image rendering is to gather and analyze semantic metadata that comes along with an image file (see Figure 1 for an example) and base further processing on the so gained information. Adding semantics has already proven to help other imaging related problems, such as object recognition [10, 13] or image retrieval [15]. The vocabulary is not controlled and users are free to enter anything that comes to mind when looking at the image. Thus, keywords can describe objects, colors, feelings and so forth. They are therefore a potentially valuable and reliable source for semantic information. A correct processing of this information has great potential to improve automatic image rendering. Figure 1. Example image with annotated keywords: trees, green, mountains, snow, quiet, blue sky, road. A first step to handle the very diverse lexicographic input from keywords is to categorize them depending on the kind of semantic information they contain. Thus, the goal of this work is the organization of semantic metadata from keywords for the specific purpose of better automatic image rendering. This work is based on real world data from a large database of photographic images [14] and the proposed methods are inspired by and evaluated with it. In this article we first discuss and propose an appropriate classification scheme for the given context. We give example images for the different classes and explain how they influence automatic image processing. Then we explain how we preprocess keywords with tools from natural language processing in order to simplify the classification task. We show how WordNet – a lexical database – can be used to efficiently classify keywords using our proposed classification scheme. We finish with an evaluation and critical discussion of the performance of the proposed classification system.