Music generation with deep learning techniques

• Main Author: Chong, Yi An
• Other Authors: Alexei Sourin
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Engineering; AI; Deep learning; Music
• Online Access: https://hdl.handle.net/10356/175381
• Institution: Nanyang Technological University

This project studies various models for deep learning music generation, and investigates a novel approach to music generation that utilises image content as an input. We use Gemini, a large language model, to generate textual captions describing the image's content and emotional tone. These captions are then fed into the existing MusicGen framework, traditionally designed for text-based music generation. While our evaluation shows promise, with generated music thematically and emotionally aligned with the corresponding image, the melodic structure remains a challenge. This suggests potential limitations in using plain text captions as input for MusicGen. Our findings pave the way for further exploration of alternative representations that could directly translate image features into musical elements. This could involve delving into image processing techniques or developing specialised music generation models that handle image data more effectively. Overall, this project demonstrates the potential of image-based music generation and highlights the need for future research in this exciting area.