Semantic communications and generative artificial intelligence for user-centric metaverse services

• Main Author: Du, Hongyang
• Other Authors: Dusit Niyato
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Engineering; Generative artificial intelligence; Semantic communications; Metaverse; Network optimization
• Online Access: https://hdl.handle.net/10356/175895
• Institution: Nanyang Technological University

Metaverse, the next frontier of the Internet, promises an immersive and intelligent world powered by advanced networks. At its core, the success of Metaverse services relies on the ability to deliver efficiency, adaptability, and satisfaction to users. Semantic Communications (SemCom) and Generative Artificial Intelligence (GenAI) are fundamental to supporting these requirements, which serve as the backbone for communications and computing paradigms. SemCom enables efficient, context-aware interactions essential for a seamless user experience, while GenAI drives the dynamic, personalized network management solutions generation to support Metaverse. In this thesis, I explore the pivotal roles of SemCom and GenAI in shaping user-centric Metaverse through innovations like Device-toDevice (D2D) information sharing, wireless sensing, and attention-aware resource allocation. These dimensions are explored in depth through subsequent chapters, showing how they collectively contribute to a more efficient, adaptable, and satisfying Metaverse experience. In Chapter 3, the focus is enhancing the efficiency of Mixed Reality (MR) technologies in Metaverse, specifically by addressing the computational constraints of Headset-Mounted Devices (HMDs). To mitigate these limitations, I introduce a full-duplex D2D SemCom strategy to reduce reliance on intensive computation through efficient information exchange. This innovative approach facilitates users sharing AI-generated content and semantic information, streamlining computational processes and enhancing the spatial coherence of computation outputs. I rigorously evaluate the performance of the full-duplex D2D communications using generalized small-scale fading models, focusing on achievable data rates and bit error probabilities. Furthermore, the chapter introduces a novel contract theoretic AI-generated incentive mechanism to enhance semantic information exchange, which, as demonstrated through numerical analysis, surpasses traditional deep reinforcement learning algorithms, including proximal policy optimization and soft actor-critic algorithms. The outcomes underscore the effectiveness and practicality of our contributions to advancing SemCom and GenAI for Metaverse. In Chapter 4, I explore the enhancement of adaptability in SemCom and GenAI for various wireless sensing tasks, such as user localization and activity detection, which are critical for user-avatar synchronization in Metaverse. Here, I introduce a novel paradigm: inverse SemCom. This approach does not extract semantic information from messages but encodes task-related source messages into a hypersource message, optimizing data transmission and storage. This chapter further details the development of an inverse semantic-aware wireless sensing framework for Metaverse, comprising three specialized algorithms for data sampling, Reconfigurable Intelligent Surface (RIS)-aided encoding, and GenAI-aided self-supervised decoding. A novel feature of this framework is the innovative RIS hardware, capable of encoding multiple signal spectrums into a single MetaSpectrum, utilizing a semantic hash sampling method for heightened encoding efficiency. Complementing this, a GenAI-aided self-supervised learning method is introduced to decode these MetaSpectrums precisely. Empirical evidence highlights a notable reduction in data volume and enhancement in accuracy for sensing tasks, underscoring the significant role of this approach in bolstering the adaptability and efficiency of SemCom for a more synchronized and responsive Metaverse. In Chapter 5, I study the significance of maximizing user satisfaction in Metaverse by enhancing personalized, immersive experiences, an endeavor traditionally constrained by the limitations of Ultra-Reliable and Low-Latency Communications (URLLC). To this end, I propose the evolution of URLLC into neXt-generation URLLC (xURLLC), incorporating personalized resource allocation to boost the Quality of Experience (QoE) significantly. This chapter elaborates on developing an optimal contract design framework, examining the interplay between Metaverse Service Providers (MSP) and network Infrastructure Providers (InP) to maximize user QoE while aligning with provider incentives. A pivotal introduction in this chapter is Meta-Immersion, an innovative metric that quantifies QoE by integrating objective performance indicators and subjective user perceptions. Furthermore, an attention-aware rendering capacity allocation scheme is developed to enhance QoE further within xURLLC by leveraging the associations between user attention and the attributes of AI-generated virtual Metaverse objects. Empirical validations using a user-object-attention dataset demonstrate that our approach can substantially improve QoE by an average of 20.1% over traditional URLLC. This represents a considerable advance in fostering a more engaging Metaverse, improving user satisfaction. In summary, this thesis constructs a comprehensive framework that positions SemCom and GenAI as supporting techniques in the evolution of Metaverse. Efficiency (Chapter 3) reduces latency and resource consumption, while adaptability (Chapter 4) ensures responsiveness to technological advancements and user preferences, collectively achieving a user-centric experience enhancement (Chapter 5). Thus, the methodologies and models introduced throughout this thesis collectively advance the theoretical understanding and practical applications of immersive Internet environments. Additionally, I explore potential future research directions, bringing insights for ongoing innovation and research in this dynamic and evolving domain.