On poisson vector graphics : displacement mapping, vectorization and color transfer

• Main Author: Fu, Qian
• Other Authors: He Ying
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2021
• Subjects: Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/145976
• Institution: Nanyang Technological University

Poisson Vector Graphics (PVG) generalize the popular Diffusion Curves (DC) by appending two new geometric primitives -- Poisson Region (PR) and Poisson Curve (PC). It is an effective graphic designing tool which extends Laplace's equation to Poisson's equation. DC is a two-sided curve with colors defined on both sides that diffuse colors over the image to produce consistent color except for sharp features along the boundaries. PC is to produce discontinuous colors across the curve, whereas PR is to style smooth shading effect among the predefined region. But DC suffers from C0 continuity. In comparison, PR is C1 continuous, that can establish smooth shading details. Furthermore, DC based vectorization works for both natural and art images quite well. But it usually requires dense curves for high-quality reconstruction, that makes post-editing difficult. Instead, PVG based vectorization converts raster image into more sparse curves, that results in easier and more convenient post-editing. PVG only extracts the curves of salient features as DCs to represent the low-frequency colors, and encode the high-frequency details in PRs, which allows to separately handle the low-frequency and high-frequency information with more flexibility. Inspired by PVG's good characteristics, we apply PVG into a few relevant fields to optimize existing methods. In the first part of this thesis, associated with the smooth nature provided by PR, we propose a novel approach of decorating 3D surfaces to composite smooth geometric details. Since PVG can produce an artistic work with sparse curves, user can directly edit synthetic geometric details and colors on the models. In our system, user can create discontinuous or sharp features by using DCs and PCs, and synthesize smooth details by PRs. PR is superior than DC since its smooth nature, and it also supports hierarchical level-of-detail editing. Comparing our method with other commonly-used technologies, their methods generally depend on global/local parameterization. Our method supports parameterization-free decoration, meanwhile diminishing the overall computational cost. In order to render PVG effectively on models, we generalize the Poisson solver from 2D domain to 3D meshes. Our solver is a local solver that designed to flexibly and versatilely render PVG. Our results manifest effectiveness of the proposed approach on both real-world and synthetic models. In the second part of the thesis, based on the property of DC that represents the salient colors of vector images, we leverage PVG to vectorize portrait images and then transfer color between the source and reference. Hereby color transfer is calculated in sparse curves rather than the entire images. First, we extract facial masks through a pre-trained neural network, and vectorize each source or reference images with sparse DCs to encode the salient (low-frequency) colors, and then calculate PRs to encode the residual (high-frequency) details. Next, we transfer DCs' boundary color between the source and reference images by applying the optimal mass transport (OMT). Finally, we render the updated PVG. Compared with existing methods that require 3D coordinate positions/normals or assume dense correspondences and alike poses between input and reference images, our method does not require pixel-to-pixel dense correspondence or 3D information, and it can process portrait images with greatly different color and postures. In the third part of the thesis, we present a simple yet effective algorithm for automatically transferring face colors in portrait videos that generalize the method of transferring color between portrait images. We extract the facial features and vectorize the faces in the input video using PVG. Then we transfer the face color of a reference image/video to the first frame of the input video by applying OMT between the boundary colors of DCs. Next the boundary color of the first frame is transferred to the subsequent frames by matching the curves. Finally, we render the video with the color-changed DCs. Thanks to our efficient DC matching algorithm, transferring colors for the vectorized video takes less than 1 millisecond per frame. Our method is particularly desired for frequent transfer from multiple references due to its information reuse nature. We demonstrate the efficacy of our method on image-to-video transfer and color swap in videos. PVG is an effective mathematic tool that can be applied into various situations to inspire better resolutions, thanks to its good properties. In this thesis, we present three relevant works. Our future direction is to explore more significative applications of PVG, associated with state-to-art technologies like deep-learning techniques.