AI foundation models

• Main Author: Chng, Joshua Yuheng
• Other Authors: Jun Zhao
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science
• Online Access: https://hdl.handle.net/10356/175068
• Institution: Nanyang Technological University

Foundation models have been a key factor in revolutionizing the way we approach various language tasks today. These models typically boast large numbers of parameters and are pre-trained on vast amounts of text data, enabling them to capture intricate linguistic patterns and relationships. However, fine-tuning these models to perform different downstream tasks has been proven to be a difficult task as most users do not have the required computational resources. Consequently, users often pass their data directly to the model owners for fine-tuning. Alternatively, model owners may share the trained weights with the users, enabling them to utilise the model for specific taskings. However, both of these solutions raise concerns over privacy and model ownership. To handle these concerns, offsite tuning has been introduced as an efficient, privacy-preserving framework for users to fine-tune foundation models offsite. The fundamental components of offsite tuning revolve around a lightweight trainable adapter and a compressed emulator. In our research, we aim to experiment with different configurations of the compressed emulator and measure the performance of offsite tuning on various downstream tasks. Our experiments have revealed 2 main findings. Firstly, performing various layer-dropping strategies to compress the emulator yields varying impacts on performance depending on the nature of the downstream tasks. Secondly, while unfreezing specific sub-layers in the initially frozen compressed emulator is generally not worth doing so given the cost-to-rewards ratio, it has resulted in slight improvements for specific datasets, giving rise to valuable insights.