Benefits from Variational Regularization in Language Models

Representations from common pre-trained language models have been shown to suffer the degeneration problem, i.e., they occupy a narrow cone in latent space. This problem can be addressed by enforcing isotropy In analogy with variational autoencoders, we suggest applying token-level loss Transformer architecture and optimizing standard deviation of prior distribution function as model parameter increase isotropy. The resulting space is complete interpretable: any given point valid embedding decoded into text again. allows for manipulations such paraphrase generation directly Surprisingly, features extracted at sentence level also show competitive results on benchmark classification tasks.