Data-efficient and privacy-enhanced knowledge discovery - Shen, Jiyuan (Nanyang Technological University, 2024)
Neural networks have undergone rapid development over the past decade, with the application of AI empowering various industrial chains. Advancing AI techniques gradually becomes a consensus in both scientific and industrial communities. Numerous new and large-scale models are emerging daily, and concurrently, this trend underscores a pressing issue: the cost of training a competent neural network model has proportionally skyrocketed. According to the latest report by OpenAI, the computational cost to achieve state-of-the-art performance in deep learning doubles every 3.4 months, whereas GPU computational power doubles only every 21.4 months, which indicates a significantly slower pace. Consequently, enhancing deep learning performance by merely increasing hardware consumption is unsustainable. On the other hand, current large-scale models are predominantly data-driven, compelling researchers to amass extensive training data. This often involves scraping nearly the entire internet for text, audio, image or video information to conduct unsupervised pre-training of large models. Take, for instance, the training of a large language model like GPT-3, which necessitates processing billions of web pages to achieve human-like text generation capabilities. Although this enormous data input empowers the model to produce highly accurate and nuanced outputs, it heightens the potential for embedding and spreading biases and personal privacy information contained in the source material, thus elevating serious privacy leakage and ethical concerns. Given the scope of such data collection, how to perform knowledge discovery through an effective and private method in machine learning presents a profound and unavoidable challenge.