Enabling control over strokes and pattern density of style transfer using covariance matrices

• Main Author: Virtusio, John Jethro C.
• Format: text
• Language: English
• Published: Animo Repository 2019
• Subjects: Image processing; Image transmission; Computer Sciences
• Online Access: https://animorepository.dlsu.edu.ph/etd_masteral/6518; https://animorepository.dlsu.edu.ph/cgi/viewcontent.cgi?article=13530&context=etd_masteral
• Institution: De La Salle University

Despite the remarkable results and numerous advancements on neural style transfer, enabling artistic freedom through the control over perceptual factors such as pattern density and stroke strength remains a challenging problem. A recent work on fast stylization networks is able to offer some degree of controllability on the pattern density by changing the resolution of the inputs. However, their solution requires a dedicated network architecture that can only accommodate a predefined set of resolutions. In this work, we propose a much simpler solution by addressing the fundamental limitation of neural style transfer models that uses the Gram matrix as its style representation. More specifically, we replace the Gram matrix with a covariance matrix in order to better capture negative spatial correlations. We show that this simple modification allows the model to handle a wider range of input resolutions. We also show that selectively manipulating the covariance matrix allows us to control the stroke strengths independently from the pattern density. Our method compares favorably against several state-of-the-art neural style transfer models. Moreover, since our approach is focused on manipulating and improving the Gram matrix, it is not dependent on any network architecture. This means that all the advancements on neural style transfer that use the Gram matrix as its style representation can directly benefit from our findings.