Pretrained Transformers for Text Ranking: BERT and Beyond

The goal of text ranking is to generate an ordered list texts retrieved from a corpus in response query. Although the most common formulation search, instances task can also be found many natural language processing applications. This survey provides overview with neural network architectures known as transformers, which BERT best-known example. combination transformers and self-supervised pretraining has been responsible for paradigm shift (NLP), information retrieval (IR), beyond. In this survey, we provide synthesis existing work single point entry practitioners who wish gain better understanding how apply problems researchers pursue area. We cover wide range modern techniques, grouped into two high-level categories: transformer models that perform reranking multi-stage dense techniques directly. There are themes pervade our survey: handling long documents, beyond typical sentence-by-sentence NLP, addressing tradeoff between effectiveness (i.e., result quality) efficiency (e.g., query latency, model index size). recent innovations, aspects they applied relatively well understood represent mature techniques. However, there remain open research questions, thus addition laying out foundations pretrained ranking, attempts prognosticate where field heading.