Skeleton-based action recognition using spatio-temporal lstm network with trust gates
Skeleton-based human action recognition has attracted a lot of research attention during the past few years. Recent works attempted to utilize recurrent neural networks to model the temporal dependencies between the 3D positional configurations of human body joints for better analysis of human activ...
Saved in:
Main Authors: | , , , , |
---|---|
Other Authors: | |
Format: | Article |
Language: | English |
Published: |
2020
|
Subjects: | |
Online Access: | https://hdl.handle.net/10356/136885 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Nanyang Technological University |
Language: | English |
id |
sg-ntu-dr.10356-136885 |
---|---|
record_format |
dspace |
spelling |
sg-ntu-dr.10356-1368852020-02-04T07:41:12Z Skeleton-based action recognition using spatio-temporal lstm network with trust gates Liu, Jun Shahroudy, Amir Xu, Dong Kot, Alex Chichung Wang, Gang School of Electrical and Electronic Engineering Engineering::Electrical and electronic engineering Action Recognition Recurrent Neural Networks Skeleton-based human action recognition has attracted a lot of research attention during the past few years. Recent works attempted to utilize recurrent neural networks to model the temporal dependencies between the 3D positional configurations of human body joints for better analysis of human activities in the skeletal data. The proposed work extends this idea to spatial domain as well as temporal domain to better analyze the hidden sources of action-related information within the human skeleton sequences in both of these domains simultaneously. Based on the pictorial structure of Kinect's skeletal data, an effective tree-structure based traversal framework is also proposed. In order to deal with the noise in the skeletal data, a new gating mechanism within LSTM module is introduced, with which the network can learn the reliability of the sequential data and accordingly adjust the effect of the input data on the updating procedure of the long-term context representation stored in the unit's memory cell. Moreover, we introduce a novel multi-modal feature fusion strategy within the LSTM unit in this paper. The comprehensive experimental results on seven challenging benchmark datasets for human action recognition demonstrate the effectiveness of the proposed method. NRF (Natl Research Foundation, S’pore) Accepted version 2020-02-04T07:41:12Z 2020-02-04T07:41:12Z 2018 Journal Article Liu, J., Shahroudy, A., Xu, D., Kot, A. C., & Wang, G. (2018). Skeleton-based action recognition using spatio-temporal lstm network with trust gates. IEEE transactions on pattern analysis and machine intelligence, 40(12), 3007-3021. doi:10.1109/TPAMI.2017.2771306 0162-8828 https://hdl.handle.net/10356/136885 10.1109/TPAMI.2017.2771306 29990167 2-s2.0-85033709940 12 40 3007 3021 en IEEE transactions on pattern analysis and machine intelligence © 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: https://doi.org/10.1109/TPAMI.2017.2771306. application/pdf |
institution |
Nanyang Technological University |
building |
NTU Library |
country |
Singapore |
collection |
DR-NTU |
language |
English |
topic |
Engineering::Electrical and electronic engineering Action Recognition Recurrent Neural Networks |
spellingShingle |
Engineering::Electrical and electronic engineering Action Recognition Recurrent Neural Networks Liu, Jun Shahroudy, Amir Xu, Dong Kot, Alex Chichung Wang, Gang Skeleton-based action recognition using spatio-temporal lstm network with trust gates |
description |
Skeleton-based human action recognition has attracted a lot of research attention during the past few years. Recent works attempted to utilize recurrent neural networks to model the temporal dependencies between the 3D positional configurations of human body joints for better analysis of human activities in the skeletal data. The proposed work extends this idea to spatial domain as well as temporal domain to better analyze the hidden sources of action-related information within the human skeleton sequences in both of these domains simultaneously. Based on the pictorial structure of Kinect's skeletal data, an effective tree-structure based traversal framework is also proposed. In order to deal with the noise in the skeletal data, a new gating mechanism within LSTM module is introduced, with which the network can learn the reliability of the sequential data and accordingly adjust the effect of the input data on the updating procedure of the long-term context representation stored in the unit's memory cell. Moreover, we introduce a novel multi-modal feature fusion strategy within the LSTM unit in this paper. The comprehensive experimental results on seven challenging benchmark datasets for human action recognition demonstrate the effectiveness of the proposed method. |
author2 |
School of Electrical and Electronic Engineering |
author_facet |
School of Electrical and Electronic Engineering Liu, Jun Shahroudy, Amir Xu, Dong Kot, Alex Chichung Wang, Gang |
format |
Article |
author |
Liu, Jun Shahroudy, Amir Xu, Dong Kot, Alex Chichung Wang, Gang |
author_sort |
Liu, Jun |
title |
Skeleton-based action recognition using spatio-temporal lstm network with trust gates |
title_short |
Skeleton-based action recognition using spatio-temporal lstm network with trust gates |
title_full |
Skeleton-based action recognition using spatio-temporal lstm network with trust gates |
title_fullStr |
Skeleton-based action recognition using spatio-temporal lstm network with trust gates |
title_full_unstemmed |
Skeleton-based action recognition using spatio-temporal lstm network with trust gates |
title_sort |
skeleton-based action recognition using spatio-temporal lstm network with trust gates |
publishDate |
2020 |
url |
https://hdl.handle.net/10356/136885 |
_version_ |
1681042432020447232 |