Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100

This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head...

Full description

Saved in:
Bibliographic Details
Main Authors: Damen, Dima, Doughty, Hazel, Farinella, Giovanni Maria, Furnari, Antonino, Kazakos, Evangelos, Ma, Jian, Moltisanti, Davide, Munro, Jonathan, Perrett, Toby, Price, Will, Wray, Michael
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2022
Subjects:
Online Access:https://hdl.handle.net/10356/162036
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-162036
record_format dspace
spelling sg-ntu-dr.10356-1620362022-09-30T06:29:55Z Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100 Damen, Dima Doughty, Hazel Farinella, Giovanni Maria Furnari, Antonino Kazakos, Evangelos Ma, Jian Moltisanti, Davide Munro, Jonathan Perrett, Toby Price, Will Wray, Michael School of Computer Science and Engineering Engineering::Computer science and engineering Video Dataset Annotation Quality This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (Damen in Scaling egocentric vision: ECCV, 2018), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the “test of time”—i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics. Published version Research at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007. 2022-09-30T06:29:54Z 2022-09-30T06:29:54Z 2022 Journal Article Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W. & Wray, M. (2022). Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal of Computer Vision, 130(1), 33-55. https://dx.doi.org/10.1007/s11263-021-01531-2 0920-5691 https://hdl.handle.net/10356/162036 10.1007/s11263-021-01531-2 2-s2.0-85117411098 1 130 33 55 en International Journal of Computer Vision © The Author(s) 2021. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/. application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering
Video Dataset
Annotation Quality
spellingShingle Engineering::Computer science and engineering
Video Dataset
Annotation Quality
Damen, Dima
Doughty, Hazel
Farinella, Giovanni Maria
Furnari, Antonino
Kazakos, Evangelos
Ma, Jian
Moltisanti, Davide
Munro, Jonathan
Perrett, Toby
Price, Will
Wray, Michael
Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100
description This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (Damen in Scaling egocentric vision: ECCV, 2018), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the “test of time”—i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Damen, Dima
Doughty, Hazel
Farinella, Giovanni Maria
Furnari, Antonino
Kazakos, Evangelos
Ma, Jian
Moltisanti, Davide
Munro, Jonathan
Perrett, Toby
Price, Will
Wray, Michael
format Article
author Damen, Dima
Doughty, Hazel
Farinella, Giovanni Maria
Furnari, Antonino
Kazakos, Evangelos
Ma, Jian
Moltisanti, Davide
Munro, Jonathan
Perrett, Toby
Price, Will
Wray, Michael
author_sort Damen, Dima
title Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100
title_short Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100
title_full Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100
title_fullStr Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100
title_full_unstemmed Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100
title_sort rescaling egocentric vision: collection, pipeline and challenges for epic-kitchens-100
publishDate 2022
url https://hdl.handle.net/10356/162036
_version_ 1745574660568776704