Human-guided cross-domain synthesis: generating virtual robotic arm imagery and videos

• Main Author: Wang, Ruofeng
• Other Authors: Wen Bihan
• Format: Thesis-Master by Coursework
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Cross-domain conversion; Image Generation; Robotic arm; Adversarial Generative Networks; Contrastive Learning
• Online Access: https://hdl.handle.net/10356/173713
• Institution: Nanyang Technological University

Currently, a multitude of interaction methods between humans and robotic arms have emerged, among which one effective strategy is to enable robotic arms to imitate human arm movements, thereby achieving intuitive operation. With technological advancements, robotic arms are now capable of learning and imitating actions by watching their videos or images. This dissertation proposes a method using cross-domain conversion and image generation technology to transform videos of human arm movements into robotic arm action videos. This method provides real robotic arms with opportunities to learn and imitate, further enabling direct interaction by mimicking human arm movements. By processing videos into frames and utilizing adversarial generative networks and contrastive learning frameworks, the mutual information between input and output domain image patches is maximized, effectively achieving cross-domain conversion. Moreover, to enhance the model’s generalization capabilities, techniques such as image masking and human skeleton keypoints detection have been introduced. This not only broadens the scope of the model’s application but also provides insights for tasks involving cross-domain conversion and opens up additional possibilities for the learning of robotic arms.