Similarity Learning Based Query Modeling for Keyword Search

In this paper, we propose a novel approach for query modeling using neural networks for posteriorgram based keyword search (KWS). We aim to help the conventional large vocabulary continuous speech recognition (LVCSR) based KWS systems, especially on out-of-vocabulary (OOV) terms by converting the task into a template matching problem, just like the query-by-example retrieval tasks. For this, we use a dynamic time warping (DTW) based similarity search on the speaker independent posteriorgram space. In order to model the text queries as posteriorgrams, we propose a non-symmetric Siamese neural network structure which both learns a distance measure to be used in DTW and the frame representations for this specific measure. We compare this new technique with similar DTW based systems using other distance measures and query modeling techniques. We also apply system fusion of the proposed system with the LVCSR based baseline KWS system. We show that, the proposed system works significantly better than other similar systems. Furthermore, when combined with the LVSCR based baseline, the proposed system provides up to 37.9% improvement on OOV terms and 9.8% improvement on all terms.