Fusing topology contexts and logical rules in language models for knowledge graph completion

Knowledge graph completion (KGC) aims to infer missing facts based on the observed ones, which is significant for many downstream applications. Given the success of deep learning and pre-trained language models (LMs), some LM-based methods are proposed for the KGC task. However, most of them focus o...

Full description

Saved in:
Bibliographic Details
Main Authors: Lin, Qika, Mao, Rui, Liu, Jun, Xu, Fangzhi, Cambria, Erik
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2023
Subjects:
Online Access:https://hdl.handle.net/10356/170544
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-170544
record_format dspace
spelling sg-ntu-dr.10356-1705442023-09-19T02:30:55Z Fusing topology contexts and logical rules in language models for knowledge graph completion Lin, Qika Mao, Rui Liu, Jun Xu, Fangzhi Cambria, Erik School of Computer Science and Engineering Engineering::Computer science and engineering Knowledge Graph Completion Information Fusion Knowledge graph completion (KGC) aims to infer missing facts based on the observed ones, which is significant for many downstream applications. Given the success of deep learning and pre-trained language models (LMs), some LM-based methods are proposed for the KGC task. However, most of them focus on modeling the text of fact triples and ignore the deeper semantic information (e.g., topology contexts and logical rules) that is significant for KG modeling. For such a reason, we propose a unified framework FTL-LM to Fuse Topology contexts and Logical rules in Language Models for KGC, which mainly contains a novel path-based method for topology contexts learning and a variational expectation–maximization (EM) algorithm for soft logical rule distilling. The former utilizes a heterogeneous random-walk to generate topology paths and further reasoning paths that can represent topology contexts implicitly and can be modeled by a LM explicitly. The strategies of mask language modeling and contrastive path learning are introduced to model these topology contexts. The latter implicitly fuses logical rules by a variational EM algorithm with two LMs. Specifically, in the E-step, the triple LM is updated under the supervision of observed triples and valid hidden triples verified by the fixed rule LM. And in the M-step, we fix the triple LM and fine-tune the rule LM to update logical rules. Experiments on three common KGC datasets demonstrate the superiority of the proposed FTL-LM, e.g., it achieves 2.1% and 3.1% Hits@10 improvement over the state-of-the-art LM-based model LP-BERT in the WN18RR and FB15k-237, respectively. Agency for Science, Technology and Research (A*STAR) This research work is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (Project #A18A2b0046). This work was also supported by National Key Research and Development Program of China (2020AAA0108800), National Natural Science Foundation of China (62137002, 61937001, 62192781, 62176209, 62176207, 62106190, and 62250009), Innovative Research Group of the National Natural Science Foundation of China (61721002), Innovation Research Team of Ministry of Education (IRT_17R86), Consulting research project of Chinese academy of engineering ‘‘The Online and Offline Mixed Educational Service System for ‘The Belt and Road’ Training in MOOC China’’, ‘‘LENOVO-XJTU’’ Intelligent Industry Joint Laboratory Project, CCFLenovo Blue Ocean Research Fund, Project of China Knowledge Centre for Engineering Science and Technology, the Fundamental Research Funds for the Central Universities (xzy022021048, xhj032021013-02, xpt012022033). 2023-09-19T02:30:55Z 2023-09-19T02:30:55Z 2023 Journal Article Lin, Q., Mao, R., Liu, J., Xu, F. & Cambria, E. (2023). Fusing topology contexts and logical rules in language models for knowledge graph completion. Information Fusion, 90, 253-264. https://dx.doi.org/10.1016/j.inffus.2022.09.020 1566-2535 https://hdl.handle.net/10356/170544 10.1016/j.inffus.2022.09.020 2-s2.0-85139295245 90 253 264 en A18A2b0046 Information Fusion © 2022 Elsevier B.V. All rights reserved.
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering
Knowledge Graph Completion
Information Fusion
spellingShingle Engineering::Computer science and engineering
Knowledge Graph Completion
Information Fusion
Lin, Qika
Mao, Rui
Liu, Jun
Xu, Fangzhi
Cambria, Erik
Fusing topology contexts and logical rules in language models for knowledge graph completion
description Knowledge graph completion (KGC) aims to infer missing facts based on the observed ones, which is significant for many downstream applications. Given the success of deep learning and pre-trained language models (LMs), some LM-based methods are proposed for the KGC task. However, most of them focus on modeling the text of fact triples and ignore the deeper semantic information (e.g., topology contexts and logical rules) that is significant for KG modeling. For such a reason, we propose a unified framework FTL-LM to Fuse Topology contexts and Logical rules in Language Models for KGC, which mainly contains a novel path-based method for topology contexts learning and a variational expectation–maximization (EM) algorithm for soft logical rule distilling. The former utilizes a heterogeneous random-walk to generate topology paths and further reasoning paths that can represent topology contexts implicitly and can be modeled by a LM explicitly. The strategies of mask language modeling and contrastive path learning are introduced to model these topology contexts. The latter implicitly fuses logical rules by a variational EM algorithm with two LMs. Specifically, in the E-step, the triple LM is updated under the supervision of observed triples and valid hidden triples verified by the fixed rule LM. And in the M-step, we fix the triple LM and fine-tune the rule LM to update logical rules. Experiments on three common KGC datasets demonstrate the superiority of the proposed FTL-LM, e.g., it achieves 2.1% and 3.1% Hits@10 improvement over the state-of-the-art LM-based model LP-BERT in the WN18RR and FB15k-237, respectively.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Lin, Qika
Mao, Rui
Liu, Jun
Xu, Fangzhi
Cambria, Erik
format Article
author Lin, Qika
Mao, Rui
Liu, Jun
Xu, Fangzhi
Cambria, Erik
author_sort Lin, Qika
title Fusing topology contexts and logical rules in language models for knowledge graph completion
title_short Fusing topology contexts and logical rules in language models for knowledge graph completion
title_full Fusing topology contexts and logical rules in language models for knowledge graph completion
title_fullStr Fusing topology contexts and logical rules in language models for knowledge graph completion
title_full_unstemmed Fusing topology contexts and logical rules in language models for knowledge graph completion
title_sort fusing topology contexts and logical rules in language models for knowledge graph completion
publishDate 2023
url https://hdl.handle.net/10356/170544
_version_ 1779156517465882624