Learning and Transferring Deep ConvNet Representations with Group-Sparse Factorization

Deep convolutional neural networks (Deep ConvNets or CNNs) have exhibited their promise as a universal image representation for recognition. In this work we explore how the transferability of such deep ConvNet representations trained on large-scale annotated object-centric datasets (ImageNet) can be further enhanced for other visual recognition tasks with limited amount of unlabeled training data. We use group-sparse non-negative matrix factorization (GSNMF), a variant of NMF, to identify a rich set of high-level latent variables built from the pre-trained Imagenet deep ConvNets that are informative across scene and fine-grained recognition tasks. The resulting architecture can itself be seen as a feed-forward model that combines deep ConvNets and two-layer structured NMF. We demonstrate state-of-the-art image clustering performance on challenging scene (MIT-67) and fine-grained (Birds-200, Flowers-102) benchmarks. The consistent superior performance of our GSNMF-CNN shows that it is more generic for novel tasks/categories compared to the deep ConvNets activations.