Learning user interface semantics from heterogeneous networks with multi-modal and positional attributes

• Main Authors: ANG, Gary, LIM, Ee-peng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Graph neural networks; transformers; attention mechanism; heterogeneous networks; multimodal; mobile application user interface; supervised learning; Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/6918; https://ink.library.smu.edu.sg/context/sis_research/article/7921/viewcontent/3490099.3511143.pdf
• Institution: Singapore Management University

User interfaces (UI) of desktop, web, and mobile applications involve a hierarchy of objects (e.g. applications, screens, view class, and other types of design objects) with multimodal (e.g. textual, visual) and positional (e.g. spatial location, sequence order and hierarchy level) attributes. We ca...