Learning user interface semantics from heterogeneous networks with multi-modal and positional attributes

• Main Authors: ANG, Gary, LIM, Ee-peng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Graph neural networks; transformers; attention mechanism; heterogeneous networks; multimodal; mobile application user interface; supervised learning; Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/6918; https://ink.library.smu.edu.sg/context/sis_research/article/7921/viewcontent/3490099.3511143.pdf
• Institution: Singapore Management University

User interfaces (UI) of desktop, web, and mobile applications involve a hierarchy of objects (e.g. applications, screens, view class, and other types of design objects) with multimodal (e.g. textual, visual) and positional (e.g. spatial location, sequence order and hierarchy level) attributes. We can therefore represent a set of application UIs as a heterogeneous network with multimodal and positional attributes. Such a network not only represents how users understand the visual layout of UIs, but also influences how users would interact with applications through these UIs. To model the UI semantics well for different UI annotation, search, and evaluation tasks, this paper proposes the novel Heterogeneous Attention-based Multimodal Positional (HAMP) graph neural network model. HAMP combines graph neural networks with the scaled dot-product attention used in transformers to learn the embeddings of heterogeneous nodes and associated multimodal and positional attributes in a unified manner. HAMP is evaluated with classification and regression tasks conducted on three distinct real-world datasets. Our experiments demonstrate that HAMP significantly out-performs other state-ofthe-art models on such tasks. We also report our ablation study results on HAMP.