DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding

Knowledge-Enhanced Pre-trained Language Models (KEPLMs) are pre-trained models with relation triples injecting from knowledge graphs to improve language understanding abilities.Experiments show that our model outperforms other KEPLMs significantly over zero-shot probing tasks and multiple knowledge-aware tasks. To guarantee effective injection, previous studies integrate encoders for representing retrieved graphs. The operations retrieval encoding bring significant computational burdens, restricting the usage of such in real-world applications require high inference speed. In this paper, we propose a novel KEPLM named DKPLM decomposes injection process pre-training, fine-tuning stages, which facilitates scenarios. Specifically, first detect long-tail entities as target enhancing KEPLMs' semantic abilities avoiding redundant information. embeddings replaced by ``pseudo token representations'' formed relevant triples. We further design relational decoding task pre-training force truly understand injected triple reconstruction. Experiments has higher speed than competing due decomposing mechanism.