KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

Abstract Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, embedding (KE) methods can effectively represent the relational facts in graphs (KGs) with informative entity embeddings, but conventional KE take full advantage of abundant textual information. this paper, we propose a unified model for Knowledge Embedding and LanguagERepresentation (KEPLER), which not only better integrate into PLMs also produce effective text-enhanced strong PLMs. KEPLER, encode descriptions PLM as their then jointly optimize modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, works remarkably an inductive KG link prediction. Furthermore, pre-training evaluating construct Wikidata5M1 , large-scale dataset aligned descriptions, benchmark it. It shall serve new facilitate research large KG, KE, The source code be obtained https://github.com/THU-KEG/KEPLER.