Spectral tensor train parameterization of deep learning layers
We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optim...
Saved in:
Main Authors: | , , , , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2021
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/6259 https://ink.library.smu.edu.sg/context/sis_research/article/7262/viewcontent/Spectral_Tensor_Train_Parameterization_of_Deep_Learning_Layers.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-7262 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-72622021-11-10T04:08:11Z Spectral tensor train parameterization of deep learning layers OBUKHOV, A. RAKHUBA, M. LINIGER, A. HUANG, Zhiwu GEORGOULIS, S. DAI, D. VAN Gool L., We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optimization problems, leading to better models and stability of optimization. We start by looking at the compact SVD parameterization of weight matrices and identifying redundancy sources in the parameterization. We further apply the Tensor Train (TT) decomposition to the compact SVD components, and propose a non-redundant differentiable parameterization of fixed TT-rank tensor manifolds, termed the Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of neural network compression in the image classification setting, and both compression and improved training stability in the generative adversarial training setting. 2021-04-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/6259 info:doi/PMLR 130:3547-3555 https://ink.library.smu.edu.sg/context/sis_research/article/7262/viewcontent/Spectral_Tensor_Train_Parameterization_of_Deep_Learning_Layers.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Artificial Intelligence and Robotics Databases and Information Systems |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Artificial Intelligence and Robotics Databases and Information Systems |
spellingShingle |
Artificial Intelligence and Robotics Databases and Information Systems OBUKHOV, A. RAKHUBA, M. LINIGER, A. HUANG, Zhiwu GEORGOULIS, S. DAI, D. VAN Gool L., Spectral tensor train parameterization of deep learning layers |
description |
We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optimization problems, leading to better models and stability of optimization. We start by looking at the compact SVD parameterization of weight matrices and identifying redundancy sources in the parameterization. We further apply the Tensor Train (TT) decomposition to the compact SVD components, and propose a non-redundant differentiable parameterization of fixed TT-rank tensor manifolds, termed the Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of neural network compression in the image classification setting, and both compression and improved training stability in the generative adversarial training setting. |
format |
text |
author |
OBUKHOV, A. RAKHUBA, M. LINIGER, A. HUANG, Zhiwu GEORGOULIS, S. DAI, D. VAN Gool L., |
author_facet |
OBUKHOV, A. RAKHUBA, M. LINIGER, A. HUANG, Zhiwu GEORGOULIS, S. DAI, D. VAN Gool L., |
author_sort |
OBUKHOV, A. |
title |
Spectral tensor train parameterization of deep learning layers |
title_short |
Spectral tensor train parameterization of deep learning layers |
title_full |
Spectral tensor train parameterization of deep learning layers |
title_fullStr |
Spectral tensor train parameterization of deep learning layers |
title_full_unstemmed |
Spectral tensor train parameterization of deep learning layers |
title_sort |
spectral tensor train parameterization of deep learning layers |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2021 |
url |
https://ink.library.smu.edu.sg/sis_research/6259 https://ink.library.smu.edu.sg/context/sis_research/article/7262/viewcontent/Spectral_Tensor_Train_Parameterization_of_Deep_Learning_Layers.pdf |
_version_ |
1770575912113274880 |