Abstract Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited unevenly distributed. This motivates research into efficient methods require ...

Abstract Probabilistic language models, e.g. those based on recurrent neural networks such as long short-term memory models (LSTMs), often face the problem of finding a high probability prediction from sequence random variables over set tokens. This is commonly addressed using form greedy decoding beam search, where limited number highest-likelihood paths (the width) decoder are kept, and at en...