MMEKG: Multi-modal Event Knowledge Graph towards universal representation across modalities

• Main Authors: MA, Yubo, WANG, Zehao, LI, Mukai, CAO, Yixin, CHEN, Meiqi, LI, Xinze, SUN, Wenqi, DENG, Kunquan, WANG, Kun, SUN, Aixin, SHAO, Jing
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Databases and Information Systems; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/7445; https://ink.library.smu.edu.sg/context/sis_research/article/8448/viewcontent/2022.acl_demo.23.pdf
• Institution: Singapore Management University

Events are fundamental building blocks of realworld happenings. In this paper, we present a large-scale, multi-modal event knowledge graph named MMEKG. MMEKG unifies different modalities of knowledge via events, which complement and disambiguate each other. Specifically, MMEKG incorporates (i) over 990 thousand concept events with 644 relation types to cover most types of happenings, and (ii) over 863 million instance events connected through 934 million relations, which provide rich contextual information in texts and/or images. To collect billion-scale instance events and relations among them, we additionally develop an efficient yet effective pipeline for textual/visual knowledge extraction system. We also develop an induction strategy to create million-scale concept events and a schema organizing all events and relations in MMEKG. To this end, we also provide a pipeline1 enabling our system to seamlessly parse texts/images to event graphs and to retrieve multi-modal knowledge at both concept- and instance-levels.