3D virtual try-on with human parsing

• Main Author: Tay, Yu Xuan
• Other Authors: Lin Guosheng
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
• Online Access: https://hdl.handle.net/10356/166088
• Institution: Nanyang Technological University

In recent years, the world saw an increase in online shopping traffic, especially during the Covid-19 pandemic, where people are stuck indoors. This led to virtual try-ons becoming increasingly popular in ecommerce platforms, because they provide a virtual fitting room space to allow shoppers to try-on new clothing. With a set up as simple as a camera and internet connection, they can see how they appear in different styles of clothing, all because of virtual try-ons. This project aims to utilise a computer vision technique known as human parsing, which is a pixel level classification task, to perform a virtual try-on. After conducting some research on already implemented virtual try-ons, this project’s objective is to implement a virtual try-on that is computationally cheap and easy to use, while being able to give the user a good idea of how they appear in different clothing. It consists of 3 stages of the try-on pipeline, the image mask generation, the thin-plate spline warp, and the 2D to 3D reconstruction. The first stage takes in the user's image and clothing image for human parsing, and prepares the necessary semantic part of both images for the second stage. The clothing mask is then warped into the shape of the user’s body and pasted onto the user, completing the 2D fit. The last step is to convert the 2D image into a 3D model for better visualisation of the fit. The 3D reconstruction uses monocular depth estimation, a depth estimation technique with a single image, followed by a point cloud generation to create a 3D space, then a poisson reconstruction of the point clouds, to produce a 3D mesh of the try-on. The virtual try-on results show decent results at the 2D stage, but the 3D reconstruction needs sharpening.