Florida International University and University of Miami TRECVID 2010 - Semantic Indexing

This paper presents the framework and results of team Florida International University University of Miami (FIU-UM) for the semantic indexing task of TRECVID 2010. In this task, we submitted four runs of results: • F A FIU-UM-1 1: KF+RERANK apply subspace learning and classification on the key framebased low-level features (KF) and use co-occurrence probability re-ranking method (RERANK) to generate the final ranked results. • F A FIU-UM-2 2: LF+KF+SF+RERANK apply subspace learning and classification on the key frame-based low-level features (KF) and shot-based low-level features (SF) separately. Then co-occurrence probability re-ranking method (RERANK) is used for both key frame based model and shot based model. Finally, a Late Fusion (LF) step combines ranking scores from each model and generates the final ranked shots. • F A FIU-UM-3 3: EF+KF+SF+RERANK apply subspace learning and classification on combined features from the selected key frame-based low-level features (KF) and shot based low-level features (SF) in the Early Fusion (EF) step. Then co-occurrence probability re-ranking method (RERANK) is used. • F A FIU-UM-4 4: SF+RERANK learning and classification based on shot based low-level features (SF). Then co-occurrence probability re-ranking method (RERANK) is used. From the results of different runs, it can be observed that F A FIU-UM-1 1 and F A FIU-UM-3 3 have better performance than F A FIU-UM-2 2 and F A FIU-UM-4 4. It implies that adding features from different sources could enhance the effectiveness of the learning and classification model and also visual features seem to be more reliable than audio features for most semantics in TRECVID 2010. The framework aims to handle several challenges in semantic indexing. For the challenge of data imbalance, Multiple Correspondence Analysis (MCA) based pruning method is able to reduce the high ratio between the number of negative instances and the number of positive instances. Meanwhile, for the challenge of semantic gap, the proposed subspace learning and ranking method has adopted a new way to select Principal Components (PCs), which spans a subspace where all instances are projected and classification rules are generated. The scores from one-class positive and negative learning models are further used to rank the classified instances. Then the co-occurrence probability re-ranking approach is utilized to improve the relevance of the retrieved shots. Please note that there is one run that adopts late fusion to combine the scores from key frame-based model and shot-based model. Evaluation results show that more efforts still need to be done to refine each module within our framework and some future directions to be explored are discussed in the conclusion section.